texts:
I'm just getting started with machine learning and I have trouble understanding how overfitting can happen in a linear regression model.
Considering we use only 2 feature variables to train a model, how can a flat plane possibly be overfitted to a set of data points?
I assume linear regression uses only a line to describe the linear relationship between 2 variables and a flat plane to describe the relationship between 3 variables, I have trouble understanding (or rather imagining) how overfitting in a line or a plane can happen?


texts: sklearn FeatureUnion vs ColumnTransformer

texts:
I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ColumnTransformer vs FeatureUnion, and they are not the same. Please advise if the following are equivalent, if not what the problem is. The data is from kaggle
https://www.kaggle.com/ronitf/heart-disease-uci

from sklearn.model_selection import train_test_split
cat_attribs = ['sex','cp','fbs','restecg','exang','ca','thal']
num_attribs = ['trestbps','chol','thalach','oldpeak','slope']
X_train,X_test,y_train,y_test = train_test_split(heart_df,y,test_size=0.25,random_state=100)

Approach #1, using column transformer

from sklearn.compose import ColumnTransformer
ct = ColumnTransformer([('oneHot', OneHotEncoder(categories='auto'),cat_attribs)
                       , ('minMax',MinMaxScaler(),num_attribs)])
ct_result = ct.fit_transform(X_train)

Approach #2, using FeatureUnion

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import FeatureUnion
from sklearn.pipeline import Pipeline
class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X[self.attribute_names].values

num_pipeline = Pipeline([('selector', DataFrameSelector(num_attribs)),
                        ('minMax',MinMaxScaler())])
cat_pipeline = Pipeline([('selector', DataFrameSelector(cat_attribs)),
                        ('oneHot',OneHotEncoder(categories='auto'))])
full_pipeline = FeatureUnion(transformer_list=[
    ('num_pipeline', num_pipeline),
    ('cat_pipeline', cat_pipeline)])
fp_result= full_pipeline.fit_transform(X_train)


texts: Relationship between batch size and the number of neurons in the input layer

texts:
Regarding LSTM neural networks, I am unable to understand the relationship between batch size, the number of neurons in the input layer and the number of "variables" or "columns" in the input. (Assuming that there is a relationship and despite seeing examples to the contrary, I cannot understand why there is no relationship)

For the sake of clarity, I am going to use an example to formulate my query.

Let's assume that the dataset contains three columns of input and one column of output. So it will be something like
input variable 1  input variable 2  input variable 3  output variable 1

From what I understand, the input layer of the LSTM network has to have 3 neurons corresponding to each of the input variables. It cannot be less or more. Even though I have seen examples like this answer (which appears to be very well described but unfortunately, I am not able to comprehend it.)

Now let us say that we have 50 rows of the above 4 columns.

That is essentially to me means that we have 50 samples.
Now if the batch size is 5, then how many input neurons we have? Is the number of neurons in the input layer independent of the batch size? The way I understand batch size is that it is the number of samples that the neural network will see before updating its weights. So let's assume if we have only three neurons in the input layer, then we will be passing the first row of input variable, followed by the second row of input variable and repeating it until the fifth row before we update the weights. How is that going to be any different than just passing the fifth row?


texts:
I have a table named app_satisfaction which has user_id, satisfaction, # of people they've invited.

I've grouped by satisfaction and found that on averge.
people in satisfaction ="BAD" group invted 2.25 people,
"GOOD" group invited 2.09 people, and
"EXECELLENT" group invited 1.89 people.

So my hypothesis is people who dislike the app are more likely to invite people since inviting people gives them free coupon and they do not like to spend their own money on app they dislike.

I have a problem, just by looking at average invite in each group it seems unreasonable to draw conclusion. Also there are more people in "GOOD", "EXCELLENT" group compared to "BAD" group.

How can I test my hypothesis? what are the approaches one might take in real world problems?


texts: Alternative methods for improved clustering separation?

texts:
I have the following labeled cluster, which is what an ideal clustering algorithm would generate:

Now, I have applied a basic K-Means clustering algorithm to the data, and the outcome is as follows:

I recognize that this is a tough problem to properly cluster because some of the classes are so similar.

But I was wondering if there are any alternative algorithms that may help me improve the separability of the clusters, and improve how well my unsupervised clustering algorithm would work on new data?


texts:
I have a list of documents which look like this:
["Display is flickering"]
["Battery charger is broken"]
["Hard disk is making noises"]

These text documents are just free text.
I have processed with tokenization, lemmatization, stop words removal, and now I want to assign tags based on a list of words. Example:
{"#display":["display","screen","lcd","led"]}
{"#battery":["battery","power cord","charger","drains"]}
{"#hard disk":["hard disk","performance","slow"]}

After text normalization I have:
["Display is flickering"] -> ["display","flicker"]

What technique is recommended to compare document: ["display","flicker"]
with my dictionary of words and return which value matches the best?
In this case I would like:
["display","flicker"] = "#display":"display"
["battery","charger","broke"] = "#battery":"charger"

Basically it compares Document A in tokens with a list B of other documents and return which document in list B with more common matches.
I'm using TF, but want to know if there are other techniques, code samples to use.


texts: Dealing with irrelevant features in dataset (Homework)

texts:
I have a specific question pertaining to one of my machine learning homeworks.

Basically, we are required to build a model that takes a 5000*10000 dataset X (5000 examples each with 10000 features), and predict Z, which is a 5000*2 Matrix. The dataset Z is synthetically generated from X, however, it only depends on two of those features from X. That is, only 2 of the 10000 features are relevant. Z has only 3 possible classes, [0,0], [1,0] and [1,1], and is not balanced. All of the features of X are sampled equally from a normal distribution, the only distinguishing feature is that Z is calculated from just 2 of them.

I thought that using PCA might be a good idea, and tried this. However, I had two problems. Firstly calling np.linalg.eig on X@X.T or X.T@X takes an unreasonable amount of time. And secondly, when I used np.linalg.eigh, it was slightly less computationally prohibitive, but many of the eigenvalues were actually very large. This makes sense to me as well since nothing actually distinguishes X except their relationship to Z.

I then built a simple neural network, linear -> tanh -> linear -> sigmoid. This descended fine, however it just learn to mirror the underlying distribution of the dataset. That is, it didn't learn which parameters of X were relevant and instead learnt to predict such that the overall error based on it (seemingly) randomly guessing was minimized.

Can anyone suggest some techniques that might help me to solve my problem?


texts:
I have the following time series data set

Each row is a unique Item, and each column shows the amount purchased per day. There are a total of 33 columns.
I'm taking the first 32 columns(leaving out the last column, which will be my target) as my training set, and the last 32 rows (leaving out the first column) as my testing set
X_train = dataset[:, :-1] # taking all columns except the last column

y_train = dataset[:, -1:] # setting the last column to be the target

X_test = dataset[:, 1:] # taking all columns expect the first column

I'm going to feed X_train and y_train in my LSTM model, and use the model to perform prediction on X_test.
Now, I wish to performing minmax scaling on dataset before performing training,  but I have some questions:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
scaled_dataset = scaler.fit_transform(dataset)

This has obvious data leakages, because the scaler is fitted with the test values.
I thought of creating 2 scalers, one for the training set, and one for the target
scaler_x = MinMaxScaler(feature_range=(-1, 1))
scaler_y = MinMaxScaler(feature_range=(-1, 1))
scaled_all_data = scaler_x.fit_transform(dataset[:, :-1])
scaled_y = scaler_y.fit_transform(dataset[:, -1:])

I'm not sure if that's the right approach. I've already tried searching for answers, but their situation is not quite like mine, or the questions are unanswered yet.
Any advice on how I should perform value scaling?


texts: Keras error on concatenating a list of tensors

texts:
I'm reproducing two versions of what I believe to be the same code, but one of them works, the other doesn't:
from tensorflow.keras.layers import Dense, Input, Lambda, concatenate
from tensorflow.keras.models import Model

inp = Input(shape=(9,))

# Version 1 (works)
out_1 = Dense(1)(Lambda(lambda x: x[:,0:4])(inp))
out_2 = Dense(1)(Lambda(lambda x: x[:,4:9])(inp))
out = concatenate([out_1, out_2])
model = Model(inp, out)
model.compile(...)
model.fit(...) ✓

# Version 2 (doesn't work)
out = concatenate([Dense(1)(Lambda(lambda x: x[:,i:j])(inp)) for i, j in [(0, 4), (4, 9)]]) # concatenating with a list comprehension
model = Model(inp, out)
model.compile(...)
model.fit(...) ✗

The error message is:
ValueError: Input 0 of layer dense_2 is incompatible with the layer: expected axis -1 of input shape to have value 4 but received input with shape (None, 5)

I'm not sure if this is a bug or an error on the code, but it looks like the concatenation is mixing the tensors when using a list comprehension. Helps appreciated :)
obs: the compile and fit functions are the same for both cases:
X = np.random.uniform(0, 1, (100, 9))
Y = np.random.uniform(0, 1, (100, 2))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X, Y)

I'm using the latest version of TensorFlow (2.3.0)


texts: My model accuracy doesn't change after first epoch

texts:
I've created a model to predict housing prices in LA, and what should be a simple regression problem, is giving me headache because the loss is just too big and my accuracy wont change.

I've already tried normalizing, changing the architecture (decreasing layers, hidden units), adding dropout, changed the loss function, batch size, epochs and my accuracy is still only 0.022

input_shape = X_train_2[0].shape

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=input_shape),
    tf.keras.layers.Dense(units=300, activation=tf.nn.relu),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dropout(0.1),
    tf.keras.layers.Dense(units=300, activation=tf.nn.relu),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dropout(0.1),
    tf.keras.layers.Dense(units = 1, kernel_initializer = 'lecun_normal', activation='linear')
    ])

model.compile(optimizer='adam',loss='mean_squared_error', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=5, batch_size=32)
model.summary()
model.evaluate(X_test_, y_test)

Training Log

Epoch 1/5 32444/32444 [==============================] - 1s
    38us/sample - loss: 90230324650039.5469 - acc: 0.0012
Epoch 2/5
    32444/32444 [==============================] - 1s 28us/sample -
    loss: 90230315396180.2031 - acc: 0.0022
Epoch 3/5 32444/32444
    [==============================] - 1s 27us/sample - loss:
    90230293267377.3438 - acc: 0.0022
Epoch 4/5 32444/32444 [==============================] - 1s 27us/sample - loss:
    90230260607518.6250 - acc: 0.0022
Epoch 5/5 32444/32444 [==============================] - 1s 28us/sample - loss:
    90230216684525.4375 - acc: 0.0022


texts:
The terms Time Series Analysis, Time Series Forecasting, and Time Series Modeling are widely used. But I don't see the clear difference in between.
Does time series analysis mean the statistical analysis of time series.
Do the time series modeling and forecasting deal with the machine learning methods of dealing with time series.
Can anyone explain the differences between these terms?


texts: About multiple objectives in Inception networks

texts:
Inception networks, in contrast to most of the networks, can have multiple outputs, from which the gradient can propagate in order to update the weights.

These outputs can have different size and solve different tasks.

Do I correctly understand, that the main point in doing so, that the
common for both output heads part of the CNN learns useful information
from both inputs simultaneously?

To get something sensible, these two tasks most likely need to correlate in a certain way, share a lot of in common. Otherwise, It looks like, the gradient updates from both heads may interfere with each other, and one does not achieve a good solution for either of the tasks.

The distance from the input to one of the heads can differ a lot from
the distance to another head. How to choose the learning rates to be
adjusted, and the proportion of the net loss for each of the
objectives?


texts: Efficient way to search list of items in a text document

texts:
I have a list of items (size ~50K) and several documents( average page per document ~10).
I am trying to find what all items are listed in each document as follows :

for document in documents_folder:
    document_text = extract_content(document)
    for item in item_list:
        if item is found in document_text:
             save(doument,item)

This takes ~5 sec per document to execute.

How can this be optimized ?
Is this the best approach ? Is there a better way to do this ?

Any suggestion is appreciated


texts: Alternative to VGAM for Zero Truncated Negativ Binomial GLM in R

texts: Is there an alternative to the vgam Package to do a zero truncated negativ Binomial GLM in R?


texts:
I would like to use tf.scatter_nd for inputs with locations = (-1, 68, 16, 16, 2) and values = (-1, 68, 16, 16).

I am having problems using scatter_nd with these inputs.

My current workaround is using tf.map_fn twice:

import numpy as np
import tensorflow as tf
with tf.Session():
    BS = 32
    locations = tf.constant(np.clip(np.random.normal(size=(BS, 68, 16, 16, 2))*112, 0, 111), dtype="int32")
    vals = tf.constant(np.random.normal(size=(BS, 68, 16, 16))*20, dtype="float32")
    def draw_lmarks(x):
        def draw_lmarks_inner(x2):
            return tf.scatter_nd(x2[0], x2[1], shape=(IMGSIZE, IMGSIZE))
        return tf.map_fn(draw_lmarks_inner, x, dtype="float32")
    imgs = tf.map_fn(draw_lmarks, [locations, vals], dtype="float32")
    imgs = K.reshape(K.max(imgs, axis=1), [-1, IMGSIZE, IMGSIZE, 1])

I'd like to get rid of the two map_fn calls.

Is there any alternative that I missed?

Edit:
Using tf.tensor_scatter_update enables me to get rid of one of the map_fn calls, and speed stuff up considerably, but i need the maximum (tf.tensor_scatter_max) which doesn't exists.
There is a tf.scatter_max tho but somehow i couldn't get that to work like expected.

With tensor_scatter_update; ie. this:

img = tf.zeros((112,112), dtype="float32")
def draw_lmarks(x):
    return tf.tensor_scatter_update(img, x[0], x[1])
imgs = tf.map_fn(draw_lmarks, [locations, vals], dtype="float32")

i get an output like this

while i need an output like this


texts: Are there enough databases for all learning tasks?

texts:
this might be a silly question but I guess the answer comes with experience in this field. I'm just wondering if today, with the internet overflowing with data and specifically with images (maybe not tagged), are there a lot of examples for learning tasks (specifically ones that takes images as inputs) without a proper database for those tasks?

I'm asking because my experience in this field is limited and I only tried to work with some more standard learning tasks that have a proper size database and I'm wondering if the need for data (or tagged data) is still an issue. So, if you can give some examples for tasks that still require a proper database of images, that will be great. Thanks.


texts: How can I use hypertools plot the dynamic graphic about mouse trajectory according to time?


texts:
I'm trying to solve a multivariate regression problem similar to PLS regression.

The problem can be described as a connectivity analysis problem where we have two regions with unknown unidirectional connections(many-to-many) and given a set of input region patterns and output region patterns, we want to infer the underlying connections.

Mathematically, the problem can be formulated as below

$Y = BX \qquad$  where $Y \in \mathbb{R}_+^{M\times N}$, $X \in \mathbb{R}_+^{L\times N}$, and $B \in \mathbb{R}_+^{M\times L}$ with $L > M >> N$

The column of $X$ and $Y$, will be a vectorized version of 2D image.

Although this would result in highly underdetermined system, I do have some prior knowledge about the pattern in input/output regions that I can incorporate in the model.

Is there a model/idea that I can use in situation like this?


texts: How to do Back Propagation Updation for below code?

texts:
Below is the code by Siraj Raval for implementing Neural Networks from Scratch. I have some doubts regarding the Code:

Why during updation he did W2 = W2 + L1.T.dot(L2_Delta). I Mean shouldn't it be W2 = W2 - L1.T.dot(L2_Delta) as we should move in the opposite direction of the gradient as in Gradient Descent
According to the fundamental equation of Backpropagation

The Update should be done as Shown in equation BP2 but in code, but the order is opposite in the code ??

  L1_Delta =(L2_Delta.dot(W2.T)) * Sigmoid(L1,Deriv = True)  # Hidden Layer Error

Similarly for Equation BP4 the Gradient

  W2 += L1.T.dot(L2_Delta)

I think It Should be :

  W2 += (L2_Delta).dot(L1.T)

# A Three Layer Model Layer 1: Input Layer
# Layer 2 = Hidden layer , Layer 3 = Output layer
#nO.OF wEIGHTS = #lAYERS - 1
import numpy as np

def Sigmoid(Z,Deriv = False):
    P = np.reciprocal(1+np.exp(-Z))
    if Deriv == False:
        return P
    else:
        return Z*(1-Z)

#Input Data
X = np.array([[0,0,1],[0,1,1],[1,0,1],[1,1,1]])
Y = np.array([[0],
              [1],
              [1],
              [0]])

np.random.seed(1)
W1 = 2*np.random.random((3,4)) - 1
W2 = 2*np.random.random((4,1)) - 1

#Training

for j in range(60000):
    L0 = X  #Input Layer
    L1 = Sigmoid(np.dot(L0,W1))  #First Hidden Layer
    L2 = Sigmoid(np.dot(L1,W2))  # Output Layer

    L2_Error = Y - L2

    if j % 1000 == 0:
        print("Error: " + str(np.mean(np.abs(L2_Error))))

    L2_Delta = L2_Error * Sigmoid(L2,Deriv = True)   #Output Layer Error
    #Now How much layer 1 contributed in the error of Layer 2
    L1_Error = L2_Delta.dot(W2.T)
    L1_Delta = L1_Error * Sigmoid(L1,Deriv = True)  # Hidden Layer Erro

    #Updates
    W2 += L1.T.dot(L2_Delta)
    W1 += L0.T.dot(L1_Delta)
print("Output After Training")
print(L2)


texts: Making predictions / Loading model in TensorFlow 2.0

texts:
I use TensorFlow/Keras on a daily basis to make predictions for a project. Everything works fine but I was getting regular warnings about the transition to TensorFlow 2.0 and I thought this week I would finally make sure my code works in the new version of the library as well. I did not encounter any problems during training or saving a model, but when it comes to making predictions I got the following warning:

  WARNING:tensorflow:Your input ran out of data; interrupting training.
  Make sure that your dataset or generator can generate at least
  steps_per_epoch * epochs batches (in this case, 10 batches). You may
  need to use the repeat() function when building your dataset.

Turns out it still does the prediction as expected, but the warning slows down the process considerably. I was able to overcome this by passing steps=1 argument to model.predict(), but this seems like a roundabout way of doing things, which was not needed in the previous version of TensorFlow.

I wonder if I am missing something trivial here. Also, it seems like TensorFlow now fails to figure out that I am doing prediction not training, which also was not an issue before. On a related note, it was probably in the documentation before as well but I never thought about the batch_size argument in model.predict() and the purpose it serves.

Update:

Now that Google Colab changed their default version to TensorFlow 2, I decided the give it another shot. Now, the code is still entirely the same, but there is an error when I try to load a model:

  WARNING:tensorflow:Error in loading the saved optimizer state. As a
  result, your model is starting with a freshly initialized optimizer.

There is an open question on TensorFlow github page on this issue:

https://github.com/tensorflow/tensorflow/issues/37968

I will update again when it is resolved.


texts: Plotting the decision boundary of different combination of 2 features from amongst a large number of features

texts:
How to plot the decision boundary of different combinations of 2 features from 107 feature data set?

# like this but it is impossible to do like for 107 features.
for pairidx, pair in enumerate([[0, 1], [0, 2], [0, 3],
                                    [1, 2], [1, 3], [2, 3]]):
        # We only take the two corresponding features
        X = iris.data[:, pair]
        y = iris.target`

source: Scikit-learn

Any suggestions for such an approach?

So that I can link it to my program:

X = array[:,[51,98]]
X = StandardScaler().fit_transform(X)
y = array[:,0]
y = y.astype(int)
# OneHot encoding"
from sklearn.neighbors import KNeighborsClassifier`

#k = 15
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

viz = DecisionViz(
    KNeighborsClassifier(10), title="Nearest Neighbors",
    features=['Feature One', 'Feature Two'], classes=['A', 'B']
)
viz.fit(X_train, y_train)
viz.draw(X_test, y_test)
print(viz.score(X_test, y_test))
viz.show(dpi = 300)

where X would be 2 out of 107 features and it would try every 2 combinations out of 107. Why would I do this, I wanted to see which 2 features were able to distinguish my 2 target classes.


texts:
I'm experimenting (read: playing around) with LSTMs on Keras.
I want to train an LSTM network so it would "track" sine waves, that is, given sine waves with different wave length, phases and lengths, it would output the rest of wave.
In a sense, a many-to-many problem.

I decided that the network would need to "track" 5 sine waves simultaneously.
I would generate random sine waves and use the 50 last data points as the required output of the NN.

My code is a follows:

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed,Lambda, Dropout
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(5, return_sequences=True))
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values

print(model.summary())

model.compile(loss='mean_squared_error',
              optimizer='adam')

def make_line(length):
    shift= np.random.random()
    wavelength = 5+10*np.random.random()
    a=np.arange(length)
    answer=np.sin(a/wavelength+shift)
    return answer

def make_data(seq_num,seq_len,dim):
    data=np.array([]).reshape(0,seq_len,dim)
    for i in range (seq_num):
        mini_data=np.array([]).reshape(0,seq_len)
        for j in range (dim):
            line = make_line(seq_len)
            line=line.reshape(1,seq_len)
            mini_data=np.append(mini_data,line,axis=0)
        mini_data=np.swapaxes(mini_data,1,0)
        mini_data=mini_data.reshape(1,seq_len,dim)
        data=np.append(data,mini_data,axis=0)
    return (data)

def train_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:] # all but last 50
        y_train = (data[:, -50:, :]) # last 50
        yield x_train, y_train

model.fit_generator(train_generator(), steps_per_epoch=30, epochs=100, verbose=1,validation_data=train_generator(),validation_steps=30)
model.save('vi2h_model.h5')

I then try to generate data and use the model to predict the continuation of a sine wave:

def data_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:]
        y_train = (data[:, -50:, :])
        return x_train, y_train

x,y= data_generator()

model = load_model('vi2h_model.h5')
pred=model.predict (x)
np.save ('pred.npy',pred)
np.save ('y.npy',y)
np.save ('x.npy',x)

When inspecting the ability of the network to predict the continuation of a sine wave (comparing y[0,:,0] to pred[0,:,0] for example, in order to compare the first sine wave of the first generated datum), see that my network performed very badly:

(Edit: legend: The predicted vs actual 50 last points in a sample sine wave (blue and orange, respectively), placed after the part of the wave used as input for the NN)

What did I do wrong? And how should I my code to correctly track sine waves?
Many thanks!


texts:
I'm reading "Hands on machine learning" by Aurelien Geron.
He stated that semi-supervised learning is:

  Some photo-hosting services, such as Google Photos, are good examples of this. Once
  you upload all your family photos to the service, it automatically recognizes that the
  same person A shows up in photos 1, 5, and 11, while another person B shows up in
  photos 2, 5, and 7. This is the unsupervised part of the algorithm (clustering). Now all
  the system needs is for you to tell it who these people are. Just one label per person,4
  and it is able to name everyone in every photo, which is useful for searching photos.

And the most important part:

RBMs are trained sequentially in an unsupervised manner, and then the
whole system is fine-tuned using supervised learning techniques.

So, here the semi-supervised learning process is to find clusters with an unsupervised learning algorithm then label those clusters with a supervised learning algorithm.

Question 1: if i want do this i have to impose that the number of clusters has to be the same as the number of labels that i have.  In the context above, if a person A is classified as B how can I handle it in the supervised step?

I always thought that supervised learning was to use a supervised algorithm and feed it with unlabeled data to improve performances, but the book of Geron says the opposite.

Question 2: do you have some site to study the semi-supervised learning in the "Geron way"?


texts: Recurrent neural network (LSTM) dimensions error

texts:
I have data in a dataframe named ddf as follows:

labels          X
L1          [1,2,3,7,8,9...]
L1          [4,2,6,9,8,7...]
...
L2          [5,6,8,9,6,3...]
L2          [7,8,5,6,9,0...]
...

There are 250 rows, 7 labels and 2000 elements in every list under X. These 2000 elements are values of a signal over a period of about 60 seconds.

I am trying to build a recurrent neural network for above data. Following is my code:

Xall = ddf['X'].values
Xall = np.array(Xall)

ydf = pd.get_dummies(ddf.drop('X', axis=1))
Yall = np.array(ydf.values)

# Split the data
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(Xall, Yall,  test_size=0.1, random_state=0)

from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
model_lstm = Sequential()
model_lstm.add(Embedding(2000, 128))
model_lstm.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(LSTM(200, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(Dense(Yall.shape[1], activation='softmax'))
model_lstm.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model_lstm.fit(X_train, Y_train, epochs=50, verbose=True, validation_data=(X_test, Y_test))

However, I am getting error at second LSTM layer:

ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2

I think this has something to do with LSTM arguments. Also are arguments of Embedding layer OK? How are both these adjusted? Where is the error coming from and how can it be solved? Thanks for your help.


texts: Item position in Gravity Search Algorithm

texts:
According to this article titled Efficient clustering in collaborative filtering recommender system: Hybrid method based on genetic algorithm and gravitational emulation local search algorithm

This is a paragraph from the article

  The current solution is chosen randomly from among the set of
  chromosomes with the lowest mass. Its space is an s-dimension space
  that is equal to the maximum number of clusters in the chromosomes
  transmitted to the GELS algorithm. The number of dimensions of the
  place of each mass is equal to s and the mass position in each dimension
  is measured by the items numbers in the cluster corresponding to that
  dimension.

From item sets of

  O={O1, O2, O3, O4, O5}.

Assuming that i have the items clustered as:

  C={C1,C2,C1,C2,C2}

So based on that paragraph i encoded the solution as

  C={C1={O1, O3}, C2={O2, O4, O5}}

Am i doing this right ?

Now currently i'm having a problem with this part

  The current solution moves in accordance with its current location
  and its new velocity in the dimension d, and is located in a new location
  in the dimension d. In fact, the new velocity value is added to the
  number value of all items in the dth cluster in the current solution

Suppose that i have a velocity value of 1
How do i add integer to a set of objects? (Ex: C2={O2, O4, O5})


texts: How to add time as a feature into word embeddings?

texts:
I have a text corpus and I'm using TfidfVectorizer. Would it be possible to cluster the resultant matrix once I concatenate tf-idf vector and time feature matrix I built([year, month, day])?
I'm also working around using word2vec for plotting a graph of similar words and I want to add time dimension for these plots.
Example: Let's say, I've five popular bug areas and for these five areas, I'm plotting similar words from the corpus using Word2Vec and Plotly. I want to have time filters on my plot such that I should be able to see these plots for different years. If my entire corpus has data from past four years then I want to visualize it for a particular year based on user selection.
I'm not sure how to achieve this. Any ideas?


texts: Would it make a difference to clean the data before using R findCorrelation()?

texts:
I'm new in data analysis area and didn't have very strong sattistical background...

Now, I'm trying to filter out those numeric columns which have high correlation. At this moment, I don't plan to use dimensional reduction algorithms such as PCA because I'm still collecting data, and there are too many columns, I hope to remove some columns first before linking different sources of data together.

Playing with PCA and machine learning models will be after linking those sources of data.

So, at this "before" stage, I'm trying to use R library caret, findCorrelation() method, but I could not find how does this method work.

Now my question is, I should use this method after data cleaning (such as, deal with missing data, outliers, data imbalance) or it doesn't matter to do it before or after?


texts: I don't understand the answer on this site for installing the network option in Orange 3.28

texts:
The instructions for using Options>Settigns>Add-ons in the post below are confusing.
Orange Error when installing Network Add On
Is there a version of the "Network" addon that will install and work with Orangev3.28 (Python3) on Win10..?
Should I check the "Conda  install add-ons with conda" box, when I provide a pip command?
What is the proper pip command, neither of the above seem to work for installing the Network addons?  What is the default setting for Options>Settigns>Add-ons so I can set it correctly since I've messed with it.
I don't want to mess up my installation of Orange by setting this preference (setting) wrong, and block installation or operation of the other add-ons.
Confused
Ron DeSpain


texts: Is there a benefit to using hadoop with only one node?

texts: I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, other then the potential of expanding to more nodes in the future? What if you have no plans of ever having more then one node? Or is there an expectation that if you are going to use Hadoop you are also going to be setting up a cluster of many nodes?


texts: How to handle negative words in word2vec?

texts: I am training a big corpus using word2vec and averaging the word vectors to get sentence vectors. What is the best way to address negative words so that negative and positive sentences are away from each other? For e.g.: "After the fix code worked" and "After the fix code did not work" should ideally give sentence vectors which are far from each other. I heard one approach is to look for negative words like "not" and negate the next word vector. Can someone please clarify if that's a good approach or can suggest a better approach?


texts: Explanation about i//2 in positional encoding in tensorflow tutorial about transformers

texts:
I was implementing the transformer architecture in tensorflow.
I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline
They implement the positional encoding in this way:
angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))

However in the paper i is not divided by 2 (i//2), is this a bug? , or why is the reason to make this operation?

thanks


texts: scikit-learn - Should I fit model with TF or TF-IDF?

texts:
I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python).

Looking at the example in the sklearn documentation, I was wondering why the LDA model is fit on a TF array, while the NMF model is fit on a TF-IDF array. Is there a precise reason for this choice?

Here is the example: http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-topics-extraction-with-nmf-lda-py

Also, any tips about how to find the best parameters (number of iterations, number of topics...) for fitting my models is well accepted.

Thank you in advance.


texts:
I am training simple Chainer based CNN to recognise MNIST samples. To each sample I add poissonian noise. For the test purpose I have always the same random seed. I restart training resetting the model for each new poisson mean.

I would expect that with increasing the mean of the poisson noise, the accuracy of the CNN should drop. Instead the behaviour is different. Indeed, when increasing the mean, the CNN takes longer and longer to train. But it either suceeds and usually gets to the target accuracy of 9X% regardless of the poisson mean, or does not train at all, staying at accuracy of ~10%. The higher the poisson mean, the bigger the probability that it will not train successfully and stay at the 10%. Could someone help me understand this behaviour?


texts:
I am dealing with a problem in which I have to label the inputs (in a sequence format) to 5 distinct classes.
The input would be like:

X = {x_1,_x_2,...,X_500}

and the output should be something like:

Y = {Y_1,Y_2,...,Y_500}

But the problem is that too many of the labels are from the first class so a sample output would have many first class labels and only a few (5 or 6 samples) related to other classes.

The classifier tends to learn to classify everything to first class and yet get higher accuracy which is not correct.

What are your suggestions?

Edit:
Loss function being used is cross entropy loss, the architecture itself is a bilstm which is applied after an embedding layer, to be more precise:

InputLayer -> Embedding -> BidirectionalLSTM -> NN -> Softmax

The input is in character format (indexes of characters) and the output for each character is of 5 distinct classes while the problem is:

Most of characters belong to first class.

PS: I hope this would leave enough information.


texts:
I am pretty fresh in machine learning so probably I made rookie mistake somewhere but I tried a lot last few days and I can't find a way to further improve my network. Hope you guys can help!

So I was asked to make "perceptron Xor gate" in Tensorflow with data that I generated myself. Since single perceptron can't solve it I used MLP. My generated train and test data looks like this:

# TRAIN SET

number_of_elements = 20000

top_right, top_left, bottom_right, bottom_left = [np.random.rand(number_of_elements, 2) for _ in range(4)]
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1

X = np.concatenate((top_right, top_left, bottom_right, bottom_left))

top_right_y, bottom_left_y = np.zeros((2, number_of_elements))
top_left_y, bottom_right_y = np.ones((2, number_of_elements))

y = np.concatenate((top_right_y, top_left_y, bottom_right_y, bottom_left_y))
y = np.reshape(y, [y.shape[0], 1])

# TEST SET

number_of_test_elements = 40000

first_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
second_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
third_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
forth_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)

top_right = np.random.rand(first_el_random, 2)
top_left = np.random.rand(second_el_random, 2)
bottom_right = np.random.rand(third_el_random, 2)
bottom_left = np.random.rand(forth_el_random, 2)
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1
X_test_v1 = np.concatenate((bottom_right, bottom_left, top_left, top_right))

top_right_y = np.zeros(first_el_random)
top_left_y = np.ones(second_el_random)
bottom_right_y = np.ones(third_el_random)
bottom_left_y = np.zeros(forth_el_random)

y_test_v1 = np.concatenate((bottom_right_y, bottom_left_y, top_left_y, top_right_y))
y_test_v1 = np.reshape(y_test_v1, [y_test_v1.shape[0], 1])

Then I made two implementations, one in Keras and one in Tensorflow. They look almost the same so I will just show Keras code:

# KERAS MODEL

model = Sequential()
model.add(Dense(64, input_dim=2, activation='tanh', kernel_initializer='normal'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(16, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dense(1, activation='sigmoid', kernel_initializer='normal'))

optimize = Adam()
learning_rate_reduction = ReduceLROnPlateau(monitor='val_binary_accuracy', patience=10, verbose=2, factor=0.5, min_lr=0.00001)

model.compile(loss='binary_crossentropy', optimizer=optimize, metrics=['binary_accuracy'])
model.fit(X, y, batch_size=200, shuffle=True, epochs=100, verbose=2, validation_split=0.33,
          callbacks=[learning_rate_reduction])

preds = model.predict_classes(X_test_v1)

real_accuracy = np.mean(np.equal(y_test_v1, preds))
print('Test classification accuracy: %.4f' % real_accuracy)

Model accuracy is around 75%, I tried to make it higher by:
- increasing train data size
- adding additional dense layers, removing layers, adding dropout, removing dropout, adding neurons, removing neurons etc
- changing activation functions, only one that I didn't change is sigmoid on outer layer, when I mess with that my results don't make sense
- messing with different optimizers, learning rates, callbacks. I didn't touch binary_crossentropy as loss function tho.
- changing bath sizes, epochs, setting shuffle to False

The problem seems pretty simple in general but I just can't pinpoint what am I doing wrong. If someone can tell me what am I doing wrong I would be really grateful. Thanks!


texts: Training a Graph model like an Artificial Neural Network

texts:
I currently have a Graph model whereby I am mapping connections of different types between entities and attributing a weight to these connections based upon my own personal experience. Also, I would like to understand the connections between these entities in relation to a particular outcome. Looking at this problem, I can't help but notice it's similarity to a typical Artificial Neural Network (ANN) and am wondering if/how I can bake some of the theory there into my model.

Let me explain further; I have a set of entities for which I have sets of connections. Each different type of connection has it's own individual weight, as shown in this diagram:

The weights in this diagram are estimated based on my own experience and can then be summed to look at the strength of a relationship between these entities. It is this strength of relationship that I want to understand in relation to a particular Output/Outcome.

I can't help but notice when looking at this particular application of Graph Theory that it bears a large resemblance to an Artificial Neural Network (ANN), as shown in this diagram;

The difference here being that whilst in the first diagram the nodes represent entities, in the second diagram the nodes represent input values. In the ANN, input values, randomized weights and the process of back-propagation are are used to help determine what the weights need to be whilst in the first diagram, these values are estimated. What I want to be able to do is to use randomized weights and the process of back-propagation to determine the weights in my Graph model that maps the connections between entities.

Does anyone have any experience of doing anything like this or could maybe at lest discuss the possibility of doing something like this?

Thanks!


texts: I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the best way to approach this, should I load into memory the 10 million rows, for example with numpy or pandas or there's a better way to load the data progressively by chunks?


texts: What is an interpretation of the $\,f'\!\left(\sum_i w_{ij}y_i\right)$ factor in the in the $\delta$-rule in back propagation?

texts:
In the $\delta$-rule which is used for error back propagation in neural networks, there is a factor $f'\!\left(\sum_i w_{ij}y_i\right)$, often written as $f'(\text{net}_i)$, which is just the derivative of the activation or transfer function evaluated at the weighted sum of the input. The derivative of $\tanh$ is bell-shaped so that values $\left|x\right|\geqslant3$ it is almost zero:

This means that units with a strong activation in either way have a more limited scope of action, where as values close to zero have more leeway. If I'm understanding it correctly, it keeps neurons that have already strongly decided one way or the other in their configuration. What is the purpose of this for weight updating? How would the networks behave differently if one would remove or change this factor?

As a reminder, the full $\delta$-rule looks like this for hidden neurons and output neurons:

$$\Delta w_{ij} = \eta\;\delta_j\;y_i$$

$$\delta_j = (\hat y_j - y_j)\;f'(\text{net}_j) \quad \text{or} \quad \delta_j = \sum\limits_k^K\left(w_{jk}\delta_k\right)\;f'(\text{net}_j)$$

where $\Delta w_{ij}$ is the change of the weight that connects the neuron $i^{\text{th}}$ neuron in the preceding layer with the $j^{\text{th}}$ neuron in the layer we are currently considering; $\eta$ is the learning rate, $y_j$ is the output of the current neuron, $\hat y_j$ is the teacher value for the $j^{\text{th}}$ output, $y_i$ is the output of the $i^{\text{th}}$, and $k$ is an index in the subsequent layer.


texts: I'm trying to fully understand the gradient boosting (GB) method. I've read some wiki pages and papers about it, but it would really help me to see a full simple example carried out step-by-step. Can anyone provide one for me, or give me a link to such an example? Straightforward source code without tricky optimizations will also meet my needs.


texts: How to associate array of class probabilities with corresponding image in keras?

texts:
I have a very basic convolutional neural net built in Keras with a TensorFlow backend. The model is based of this Kaggle kernel full model. The input training data are 256x256 images with a corresponding set of class labels in a csv file. For example, here are the class labels in the csv file:

image_name  |  tags
----------------------------------------
train_0     |  class1 class2 class3
train_1     |  class2
train_2     |  class3
and so on ...

The following code shows how I am building, training, and predicting results.

# Build model
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=(32, 32, 3))) # Originally (32, 32, 3)

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='sigmoid')) # Originally 17

model.compile(loss='binary_crossentropy', # We NEED binary here, since categorical_crossentropy l1 norms the output before calculating loss.
              optimizer='adam',
              metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train,
          batch_size=128,
          epochs=4,
          verbose=1,
          validation_data=(x_valid, y_valid))

# Predict
p_valid = model.predict(x_valid, batch_size=128)

You can see the predicted results are an array of class probabilities:

p_valid
Out[29]:
array([[2.4571007e-01, 1.4929530e-02, 9.3575776e-01],
       [2.6206359e-01, 1.2429485e-02, 9.5172155e-01],
       [3.3679003e-01, 2.8344743e-02, 8.5209453e-01],
       ...,
       [8.2605546e-03, 2.0513092e-07, 9.9999821e-01],
       [8.2605546e-03, 2.0513092e-07, 9.9999821e-01],
       [8.2605667e-03, 2.0513131e-07, 9.9999821e-01]], dtype=float32)

How do I know which probability array is corresponding to which image?


texts: Is there a way I can access just the vocabulary list of pre-trained vectors for word2vec and GloVe? I do not need the entire n-dimensional embeddings.


texts: Named Entity Recognition for Sesotho sa Leboa (Sepedi) one of the South African Official Language

texts: I'm looking for model to used to develop NER for Sepedi?


texts:
I'm trying to tune hyperparameters with bayesian optimization. It is a regression problem with the objective function: objective = 'reg:squaredlogerror'
$\frac{1}{2}[log(pred+1)-log(true+1)]^2$
My dataset consists of 20k vectors, each vector has length 12 (twelve features). Every vector has a corresponding Y value.
I want to find the set of hyperparameters that minimize the loss function.
This is how it is implemented in code:
def evaluate_model(learning_rate, max_depth, nr_estimators, min_child_weight, min_split_loss, reg_lambda):
    model = get_model(learning_rate, max_depth, nr_estimators, min_child_weight, min_split_loss, reg_lambda)
    model.fit(X_train, Y_train)
    pred = model.predict(X_val)

    error = np.array([])
    for i in range(len(pred)):
        prediction = np.maximum(pred[i],1)
        error = np.append(error, (1/2)*(np.log(prediction+1)-np.log(Y_val[i]+1))**2)
    err = np.mean(error)
    return -err

My question is if anyone has any problem with how I've constructed the evaluate_model function. Do this optimize the squared log error when bayesian hyperoptimization is being implemented? The maximum(pred[i],1) is there in case a negative prediction is produced. Also, I get bad results even after the hyperparameter optimization.
These are the hyperparameters I evaluate:
pbounds = {'learning_rate': (0,1), 'max_depth': (3,10), 'nr_estimators': (100, 5000), 'min_child_weight': (1,9), 'min_split_loss': (0,10), 'reg_lambda': (1,10)}

The optimization is ran for 100 iterations and 10 init points. The package I've used for the bayesian optimization is bayes_opt


texts: How to best accomplish high speed comparison of like data?

texts:
I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break.  I would like some input on how to speed things up.

I have multiple datasets of information in a SQL database.  The database is vendor-designed, so I have little control over the structure.  It's a sql representation of a class-based structure.  It looks a little bit like this:

Main-class table
 -sub-class table 1
 -sub-class table 2
  -sub-sub-class table
 ...
 -sub-class table n

Each table contains fields for each attribute of the class.  A join exists which contains all of the fields for each of the sub-classes which contains all of the fields in the class table and all of the fields in each parent class' table, joined by a unique identifier.

There are hundreds of classes. which means thousands of views and tens of thousands of columns.

Beyond that, there are multiple datasets, indicated by a field value in the Main-class table.  There is the production dataset, visible to all end users, and there are several other datasets comprised of the most current version of the same data from various integration sources.

Daily, we run jobs that compare the production dataset to the live datasets and based on a set of rules we merge the data, purge the live datasets, then start all over again.  The rules are in place because we might trust one source of data more than another for a particular value of a particular class.

The jobs are essentially a series of SQL statements that go row-by-row through each dataset, and field by field within each row.  The common changes are limited to a handful of fields in each row, but since anything can change we compare each value.

There are 10s of millions of rows of data and in some environments the merge jobs can take longer than 24 hours.  We resolve that problem generally, by throwing more hardware at it, but this isn't a hadoop environment currently so there's a pretty finite limit to what can be done in that regard.

How would you go about scaling a solution to this problem such that there were no limitations?  And how would you go about accomplishing the most efficient data-merge?  (currently it is field by field comparisons... painfully slow).


texts: Universal function approximation with fixed values (as vector or matrix)

texts:
I was thinking about way to represent/approximate universal function and came up with the idea that a plain fixed numbers could be used to represent pretty much any function on a fixed interval.

I wonder how such approach called. And if it's used in logistic regression, genetic algorithms etc.?

The good thing is that it's universal, you can extend it to multiple dimensions and you don't have to make any assumption about function if it should be polynomial or trigonometric etc.

I would like to know more about it. But can't find much.


texts: Why KMeans.score() gives very high value?

texts:
I'm working on Udacity's Identify Customer segments project. My problem now is that after applying StandardScaler on the dataset after procedures of cleaning, I used PCA(n_components=36).fit_transform(dataset). The problem came when I applied KMeans with different n_clusters to plot the elbow chart to choose the best n_clusters, the scores were in million as shown in this screenshot. What's the problem? any help, please!!

here's the code I used
# Apply feature scaling to the general population demographics data.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
azdias_transform = scaler.fit_transform(azdias_clean)
# Re-apply PCA to the data while selecting for number of components to retain.
pca_36 = PCA(n_components=36)
pca_azdias = pca_36.fit_transform(azdias_transform)
# Applying KMeans
from sklearn.cluster import KMeans
def kmeans_clustering(data, n_clusters):
    kmeans = KMeans(n_clusters=n_clusters)
    kmeans_fit = kmeans.fit(data)
    score = np.abs(kmeans_fit.score(data))
    return score
ls_score = []
clusters = list(range(3,30,3))
for i in clusters:
    ls_score.append(kmeans_clustering(pca_azdias, i))

and here's the project repo


texts: What is the best way to cluster this kind of data?

texts:
I have data that looks like this:

The chart on the left is the trend and the smaller chart on the right is the box plot showing the distribution of means. Each color is the output of a particular tool. I dont need to do an ANOVA and Tukey tests on it because it can easily be seen that there are statistically significant differences between the tools.
However, I would like to group/cluster the tools based on their mean.
For example, visually speaking, one can separate all the tools into 4 groups. 4 tools with a mean around 32, 12 tools with a mean around 30, 5 tools whose variance is quite high, and whose mean don't really fall into a particular group and the rest whose mean is around 30.
Update:
I have used the k-means clustering technique and the Hierarchical Agglomeration techniques. However, I have to specify the number of clusters in advance. Is there any clustering technique where I don't have to specify the number of clusters a-priori?


texts: Neural network for peak identification problem not training successfully

texts:
I'm trying to build a program that can identify peaks in a signal.

It's my first attempt at machine learning so I'm starting with a toy problem where I'm generating a gaussian curve of random width and position with a small amount of added noise and trying to identify what part of the signal looks like a peak but I'm facing some issues. I'm using Theano+Lasagne and adapted my code from the MNIST.py example. Here's an example of what I feed to my code and what I get as a result:

My idea is to feed the signal (1D vector of 512 values) to a simple neural network composed of 5 dense layers (3 linear followed by 2 sigmoids) of 128 neuron each. And training it on tens of thousands of generated examples.

But the issue I'm facing is that no matter what I try regarding weight initialization or learning parameters I always converge to the same result : a network that seem to only spit out the average value of the "goal output" of all the examples it was trained with.

I would assume that given the structure of my neural net and what data I'm feeding to it, it would easily converge to something that does a bit of averaging and thresholding to give me something reasonably close to the expected output.

Is that foolish to assume ? Or am I missing something important ?


texts:
I am working on a problem to identify anomaly in Network. I am stuck at how to handle the following issues
1. VPN Land Based Violation (Login from Multiple locations within unrealistic situation
2. VPN Timebased violation- Multiple logins at the same time
I know we can figure this out using a simple sql query.. but How do we deploy the model in production to find these? Like how do we deploy a model to find out if the person has logged in Asia and in the next hour login from Europe.
Aprreciate any help.

I will collect data from Spark using Kafka stream and then will run a machine learning model to detect Anomalies. I will be building a behaviour profile for each user. I need help with How do i actually set it up so that the data can be collected from spark continuously and exported to cloud or should I use Spark MLlib to run the model and then export the results(or threats) to a database which can be viewed by Security? or Just use Spark MLlib and setup such that alerts can be sent to the network team. New to Spark and cloud sorry if I am not clear
Thank you


texts:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Importing Dataset

dataset = pd.read_csv('C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 8 - Deep Learning/Section 39 - Artificial Neural Networks (ANN)/Churn_Modelling.csv')
print(dataset)
X = dataset.iloc[:, [3, 13]].values
Y = dataset.iloc[:, 13].values
print(X)

#Categorical Data

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder1 = LabelEncoder()
X[:, 1] = labelencoder1.fit_transform(X[:, 1])
try:
    labelencoder2 = LabelEncoder()
    X[:, 2] = labelencoder2.fit_transform(X[:, 2])
except IndexError: pass

#Dummy Variable
onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()

#Splitting the dataset into training set and test set

from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=0)
print(Y_test)

#Feature Scaling

from sklearn.preprocessing import StandardScaler
feature_scaling = StandardScaler()
X_train = feature_scaling.fit_transform(X_train)
X_test = feature_scaling.transform(X_test)
print(X_train)

#Importing Keras libraries and packages

import keras
from keras.models import Sequential
from keras.layers import Dense

#Initialising the ANN

classifier = Sequential()

#Adding the input layer and hidden layer
classifier.add(Dense(input_dim=11, units=6, kernel_initializer='uniform', activation='relu'))

#Adding the second hidden layer
classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu'))

#Adding the Output Layer
classifier.add(Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))

#Compiling the ANN(Applying Stochastic Gradient)
classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

#Fitting ANN to training set
classifier.fit(X_train, Y_train, batch_size=10, nb_epoch=100)
# predicting the test result
Y_pred = classifier.predict(X_test)
Y_pred = (Y_pred > 0.5)

# Making the confusion Matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(Y_test, Y_pred)
print(cm)

I am trying to fit the ANN model to my dataset, after preprocessing of data and importing the keras library, the following error comes during model fitting.

Traceback (most recent call last):
  File "<input>", line 2, in <module>
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\keras\engine\training.py", line 952, in fit
    batch_size=batch_size)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\keras\engine\training.py", line 751, in _standardize_user_data
    exception_prefix='input')
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\keras\engine\training_utils.py", line 138, in standardize_input_data
    str(data_shape))
ValueError: Error when checking input: expected dense_1_input to have shape (11,) but got array with shape (3,)

I cannot understand what is wrong, please help.


texts: Column sum in SPSS (with filter and grouped by date)?

texts:
device      date     act power 1     react power 2
-------------------------------------------------
M1         05-02            2              3
M2         05-02            4              2
M3         05-02            3              4
M1         06-02            1              2
M2         07-02            3              4
                     -------        -------
                    need sum       need sum

Say that I only need the sum of M1 and M2 from that table. How could I add a variable that contains the sum of power group by date and device?

I don't know if it is desired to have something like this? Or how can I use descriptive statistics to filter away the other devices just to have M1 and M2 only.

device      date     SUM (actpower1)     SUM(reactpower2)
-------------------------------------------------
M1+M2      05-02            6              5

```


texts: What to do with large number of collinear variables?

texts:
I have this time-series dataset that has 63 features, out of which 57 were manually engineered. While checking for collinearity, I get this correlation matrix:

As can be seen there are a number of variables that are correlated/collinear. The ones that are deep red certainly need to be removed, but what about the ones on the bluer range? How do such variables (on the negative range of collinearity) effect regression models?
Also, I ran a recursive feature extraction process from sklearn.feature_extraction module and it has recommended me 39 features to be the best (at default settings). Is RFE the best strategy while dealing with such features?


texts: Toeplitz matrix in convolution neural network problem

texts:
Instead of multiplying the kernel with input vector iteratively, the convolution operation could be written as matrix multiplication. Infact this is how convolution operation is implemented efficiently in machine learning libraries. That is, y = Ax, where y belongs to R^4, and x R^6 are column-vectors, and A belongs to R^4*6 is the Toeplitz matrix whose elements are arranged in special order using the elements of kernel k.
Does simply transposing the matrix A and multiplying with y would recover x? and How can I construct matrix A?
I would be thankful if you help me as a beginner prospective


texts: best approach for CNN training with multiple subcategories and one category

texts:
I need to classify pictures into 2 categories: approved and rejected.
Rejected category has different type of images which are not allowed (subcategories), for example nude or gore or anime etc.

What approach in training the CNN will be better or they are equal for the CNN:

To have output final layer with 2 categories (neurons) -
approved and rejected
To have a lot of neurons in the final layer, one for each "subcategory" (and feed NN with corresponding labels), and later when doing actual inference just manually
aggregate them to rejected category?

I'm not sure if using the first approach CNN can easily apply OR operation for such a different subcategories, thus that training will be effective. Just feel that multi-categories approach is easier for NN. Is there any approved science behind it?


texts: Ordinal Classification As Cost Function - In Keras or Tensorflow

texts:
I am having a hard time implementing/using an ordinal classification cost function for my Convolutional Neural Network in Tensorflow.

Any implementations/documentation/repository on that which you know of?


texts:
I am new in ML field and I got this question when classifying the DEAP data with sklearn. There is my data set description.

The whole data is a (1280*322) matrix，which the last two columns are the labels and the rest of it is the feature（1280*320).  Because the output of the labels has three categorical values, I try to predict this problem with RandomForestClassifier. However, the mean accuracy of the result is just 0.222656250. Would you give me any clues about this problem? Thanks for your patience. the detail code as follows:

RandomForestClassifier

import scipy.io
from sklearn.datasets import make_classification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.utils import shuffle
import numpy as np

def mydata():
    data = scipy.io.loadmat('/home/jdang/Documents/RenKaimeng/EEGfile/Classification-for-DEAP/whole.mat')

    # initialization

    x_vals = data['all_data'][:,0:320]

    y_vals_Valence = np.array([0 if each=='Neg'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,320]])
    y_vals_Arousal = np.array([0 if each=='Pas'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,321]])

    Y = np.vstack((y_vals_Valence, y_vals_Arousal)).T

    DEAP_x_train = x_vals[:-256] #using 80% of whole data for training
    DEAP_x_test = x_vals[-256:]  #using 20% of whole data for testing
    DEAP_y_train = Y[:-256]
    DEAP_y_test = Y[-256:]

    forest = RandomForestClassifier(n_estimators=100, max_depth=2,random_state=1)
    multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
    multi_target_forest.fit(DEAP_x_train, DEAP_y_train)

    print("the result of MutliOutput-classification-using-RandomForest ")
    print('The mean square error %10.9f ' % np.mean((multi_target_forest.predict(DEAP_x_test)-DEAP_y_test)**2))  # The mean square error

    print('the mean accuracy on the given test data and labels %10.9f'% multi_target_forest.score(DEAP_x_test, DEAP_y_test))


texts: A question about SAMME algorithm (Adaboost)

texts:
According to this article (which references the original paper), this is the SAMME algorithm for multiclass classification using Adaboost:

I would like to understand what is this term in step 2.d which multiplies the $\alpha^{(m)}.$ It also appears in step 2.b. I think it has something to do with the samples that are missclassified, but i can't understand what it does exactly.
Also, i would appreciate some explanation about the Output (prediction) of this algorithm.
Thanks!


texts: How to find farthest data points from a predefined cluster in a data set with Python?

texts:
I have a data set where certain rows are labeled as one class (and interpreted as distinct cluster #1 as such), but other points are either unlabeled or ambiguous. Hence I want to figure out which unlabeled data points lie farthest from cluster #1 by sorting them by their respective distance from cluster #1 (more precisely, from the closest point of cluster #1 to the respective unlabeled points).
My first idea would to create a similarity matrix between and calculate the closest distances per unlabeled points from this, but somehow this seems a but clumsy, is there a more elegant/effective way?
(I used to use sklearn for similar tasks, but as far as I know, unsupervised clustering algos don't explicitly provide this kind of specific information.)


texts:
I'm trying fit my data , but I couldn't fit it.Data set

0.50,0
0.75,0
1.00,0
1.25,0
1.50,0
1.75,0
1.75,1
2.00,0
2.25,1
2.50,0
2.75,1
3.00,0
3.25,1
3.50,0
4.00,1
4.25,1
4.50,1
4.75,1
5.00,1
5.50,1

and my code

data = np.loadtxt('dat', delimiter=',',dtype=None);

x=data[:,0:1];
y=data[:,1].reshape(x.size/x[0].size,1);
a=np.ones(shape=(y.size,x[0].size+1));
a[:,1:2]=x;
q=np.ones(shape=(a.shape[1],1));
alpha=0.003

for i in range(500000):
    h=1/(1+np.exp(-np.dot(a,q)))
    for j in range(q.size):
        q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j]);
plt.axis((-1,10,-1,5))
plt.plot(x,y,'x',x,h);
plt.show();

So I tried different learning rates(alpha),tried different number of iterations but my fitting data is looking like this
but it's should looks like this enter link description here

What am I missing? Is there any logical error or something like that? Thanks for your deal.


texts: Why is np.where not returning '1'? Only returns '0'

texts:
This code should return a new column called orc_4 with the value of 1 if the value of the row in df['indictment_charges'] contains 2907.04 or 0 if not.
Instead it is returning all 0's
for index, item in enumerate(df.indictment_charges):
    s = '2907.04'
    if s in str(item):
        df['orc_4'] = np.where(item == s, 1, 0)

Why won't it return 1?
Example output for df.indictment_charges:
['2903.112907.022907.042907.04']


texts: Which term is correct Datafication or Datification?

texts:
I have recently started reading Introduction to Data Science: A python approach to Concepts, Techniques, and Applications  and taking notes on Data Science.
Chapter 1 repeatedly uses the term Datification(the process of rendering into data aspects of the world that have never been quantified before). I could not find the word in the dictionary and the web shows some results but with different spellings Datafication(a technological trend turning many aspects of our life into data).
I am wondering which one is correct?


texts: What (probabilistic models) can only output decisions when they are certain?

texts:
I'm basically looking for approaches, models, algorithms for the following situation (a fault diagnosis problem):

I have an input set $\{x_i\}_{i \in \{1..m\}}$ with $n$ binary features of cases (think of "faults" or "alarms" that fired) and $k$ classes. Each case $x_i$ can belong to at least one class and at most $k$ (so I'm dealing with multi-label classification).

Now some relations in the data set are utterly boring/uninformative (say, feature $a$ says "Mechanical Error occurred" and label $b$ means "Mechanical Error fixed").
But even more generally, whenever $x_a = 1$, I see all kinds of labels, i.e., $a$ is not predictive. Put differently, the relation is not "functional".

Other input features $c$ might have a much more "functional" nature, such that whenever $x_c = 1$, I can easily deduce $y_d = 1$.

For instance my training set could look something like this:

$ [0, 1,0] \mapsto \{4, 1\}$

$ [0, 1,0] \mapsto \{2\}$

$ [1, 0, 0] \mapsto \{1\}$

$ [1, 0, 0] \mapsto \{1\}$

So, knowing $[0,1,0]$ is not really informative whereas $[1,0,0]$ tells me (with high confidence) that the label 1 is active.

I'm looking for the latter pairs, a classifier that only extracts meaningful pairs and ignores uninformative inputs.

Could you point me to relevant techniques / keywords? Thanks a lot!


texts:
I was curious to see if one can use a cost function on a set of data points to find the "optimial minimum" solution for any given set of data.

I know for a regular set of data that is clustered symmetrically following a straight regression line is easy to find the proper cost function

Example:

but what if the data makes a funky shape like that of x^2 or 3*x^3 or etc...

As you can see that the the second graph is a lot more noisy than the first.

Then what do you do? is it the same process or is it different?
Can you find an optimal Minimum for any set of data points?

I know the cost function is using the residuals to form its best fit but I was just curious if it can do with any set of data.


texts: Metrics for Evaluating Performance of Logistic Regression

texts:
I built a Logistic Regression model and I would like to evaluate the performance of the model. I would like to understand its evaluation metrics.

What do the metrics Sensitivity, Specificity, False Positives Rate, Precision, Recall, and Accuracy tell us about this model?


texts: Multi-label classification based on single-label dataset

texts:
I'm looking for a solution to detect different moods/styles expressed by an image. Unfortunately, there is no multi-labeled dataset for this task.

The scenario of defining a multi-label classification model based on single labeled data doesn't seem too absurd to me but still I couldn't find any publications or other sources addressing this problem.

So I'm very thankful for any hint how scenarios like this could be tackled (e.g. deriving a multi-labeled dataset from available single label datasets in a weakly supervised manner).


texts: Methodology for parallelising linked data?

texts:
If I have some form of data that can have inherent links to all other data in the set but I wish to parallelise out this data in order to increase computation time or to reduce the size of any particular piece of data currently being worked on, is there a methodology to split this out into chunks without reducing the validity of the data?

For example, assume I have a grid of crime across the whole of a country. I wish to treat this grid as a heat map of crime and therefore "smear" the heat from a crime to nearby points.

However, the time to calculate this is too long if I try to do it across the whole of the country. Therefore, I want to split this grid out into manageable chunks.

But if I do this then high crime areas on the edges of the chunks on this grid will not be "smeared" into nearby areas. I don't want to lose this data validity.

What, if any, is the methodology to solve this linking of data in parallelisation?


texts: Conditionally replace dataframe cells with value from another cell

texts:
I have a couple pandas dataframe questions. I would like to replace the values in only certain cells (based on a boolean condition) with a value identified from another cell. I have defined the dataframe from an imported text file, which returns a dataframe with column headers 'P' and 'F' and values in all of the cells. I want to replace only those cells in column 'F' where the corresponding 'P' entry in the same row is less than a defined variable, Plithos. The new value for all of the replaced cells is defined as Fmax, which is the value of 'F' when 'P' in the same row == Plithos:

Plithos = 5.0

Fmax = df.loc[(df['P']==Plithos),'F']

The above part seems to work. The Fmax value returned is the correct one from the table. But when I try to replace the appropriate values using the code below, the right cells are identified but the new entries are all NaN:

df.loc[(df['P'] < Plithos),'F'] = Fmax

I'm not sure why this is happening, or what I should change in the syntax to fix it? Any help is appreciated!

Second question: Ideally, I would also like to define the condition for Fmax to be the value found in the row where P is equal to or less than Plithos, but the closest possible (so, the Price is Right match, I guess?). That way if the user input spreadsheet doesn't have a perfect matching value for Plithos in their table, the code still works. I assume pandas doesn't have a Price is Right boolean built in, though...


texts: Training with many CPU cores doesn't improve performance

texts:
I ran my job on a computing cluster: first with 1 node / 4 cores, then with 2 nodes / 32 cores. But the training time is pretty much exactly the same for both of them! 67 seconds per step.

I am trying to fine-tune GPT-2 for my text dataset (chat logs).

What can I do to get a performance increase corresponding to the increase in processing power with CPUs?


texts: Could someone please offer me some guidance on some kind of particular, SPECIFIC project that I could attemp, to "get my feet wet, so to speak"

texts:
I am COMPLETELY new to the field of Data Science, mainly because every employer I have worked for, simply COULDN'T sell any customers anything that would use techniques learned in this field.

Of particular interest to me is machine learning/Predictive Analysis.

I have attempted many "test projects" myself, but I seem to NEED some sort of outside "catalyst" to tell me a specific goal, and a specific set of guidelines, when I am trying to learn something.
Otherwise, I tend to lose focus, and jump from one interesting topic to the next, without ever gaining any experience.

Thank you!!


texts: Keras Reproducibility Problem on Jupyter notebook

texts:
I developed a model with Keras. However, every time I run the model from the beginning, I get different score values. I typed the code below to provide "reproducibility" but unfortunately, I'm still getting different results. Is there any other method you have tried to achieve "Reproducibility"?
import pandas as pd
import random
import numpy as np

seed = 0
random.seed(seed)
np.random.seed(seed)

import os
os.environ['PYTHONHASHSEED'] = str(seed)
os.environ['TF_DETERMINISTIC_OPS'] = '1'

import tensorflow as tf
tf.random.set_random_seed(seed)
tf.set_random_seed(seed)


texts: Keras ModelCheckpoint Callback returning weights only even though both save_best_only & save_weights_only are set to False

texts:
I am trying to understand what happens here when I use the Keras ModelCheckpoint callback without setting either save_best_only & save_weights_only to True.
i.e.
from keras.callbacks import ModelCheckpoint
mc = ModelCheckpoint(filepath = './checkpoint/to/save.h5')

According to Tensorflow, both save weights and save best are both set to False by default.
tf.keras.callbacks.ModelCheckpoint(
    filepath, monitor='val_loss', verbose=0, save_best_only=False,
    save_weights_only=False, mode='auto', save_freq='epoch',
    options=None, **kwargs
)

So when none are initialized to True, what gets saved?


texts: How do I use this depth first search code to obtain a topological sort?

texts:
Problem: I need to implement a topological search using the following depth first search code.

Note: The original code comes from here, and this is a problem given in at the end of the chapter.

I'll be honest. On a coding level, I feel pretty lost. I've added line comments to help show what I think each line is doing. While I'm fairly new to the depth first search, I understand how they work on a functional level (as opposed to my limited knowledge of their implementation). I've attempted to add the start vertices and finish times to a list so that I could then sort them later on, but I've had trouble returning that list. So that I could then sort the vertices in decreasing order of finishing time.

Note that while some solutions seem to use a stack, this code does not state the use of a stack explicitly. Rather, the stack is implicit in the recursive call to dfsvisit.

from pythonds.graphs import Graph
# Definitions
# discovery time == iterations it took for the program to find the vertex and turn it gray
# finish time == iterations it took for the program to turn the vertex black
# pred == predecessor indicator

class DFSGraph(Graph):
    def __init__(self):
        super().__init__()
        self.time = 0                       # self has an attribute 'time' (counter) that initiates at 0

    def dfs(self):
        for aVertex in self:
            aVertex.setColor('white')
            aVertex.setPred(-1)
        for aVertex in self:
            if aVertex.getColor() == 'white':
                self.dfsvisit(aVertex)

    def print_graph(self):
        for key in sorted(list(self.vertices.keys())):
            print(key + str(self.vertices[key].neighbors + " " + str(self.vertices[key].dis)))

    def dfsvisit(self,startVertex):                     # Initiate the visit vunction of the current object (self) at the starting vertex (startVertex)
        finish_times = []                                       # Instantiate the a list to keep finish times for each node
        startVertex.setColor('gray')                    # Set the color of the starting vertex to 'gray' (discovered)
        self.time += 1                                  # Increment the timer
        startVertex.setDiscovery(self.time)             # Assign the discovery time to the current vertex (startVertex)
        for nextVertex in startVertex.getConnections(): # Begin cycling through the connected vertices (nextVertex) of startVertex
            if nextVertex.getColor() == 'white':        # If a vertex (nextVertex) with the attribute color of white is found, then do the following:
                nextVertex.setPred(startVertex)         # Set the predecessor indicator of the next vertex as the current vertex (ie if B is white and connected to A, set B's predecessor as A)
                self.dfsvisit(nextVertex)               # Recursively calls itself with the next vertex until the color of the next vertex is no longer white (i.e. all have been explored)
        startVertex.setColor('black')                   # Set the current vertice's color to black (explored)
        self.time += 1                                  # Increment the timec counter
        startVertex.setFinish(self.time)                # Assign the finish time to the current vertex (startVertex)
        finish_time = [startVertex, startVertex.setFinish(self.time)]  # append vertex and finish times to finish_time list
        return finish_time                                             # return the finish_time list

Again, I'm hoping to understand how to implement the topological search either within this code.


texts: I am beginner in natural language processing and my goal is to find a way to score sentences based on their emotional tension. More specifically, I would like to know to what degree a sentence transmits wish, hate, or fear. I looked at some studies on sentiment analysis, but haven't seen any relevant result. Most of the them are on negativity or positivity of a sentence.


texts: Can we use boosting algorithms like Adaboost and gradient boosting with only one classifier

texts: I have been working on ensemble learning and I came across this doubt that unlike other ensemble learning algorithms like voting classifier a can we only use one classifier with boosting.


texts:
I have end-of-day time-series dataset with following data

    Ticker  Date    Open    High    Low     Close   Volume

I have calculated end-of-day closing returns (Close_t/Close_t-1) -1 at time t, ratio of High and Low. Each dataset closing returns and High/Low is of size N x p, where  N are dates and p are tickers and N>p. Hence closing returns dataset looks like this

Date  Ticker1  Ticker2 --- until p
  --
  --
  --
 until N
and similarly high/low ratio, where Ticker is the company code.
I want to forecast future closing return using any simple model (e.g., Linear Regression). My question is how do I perform analysis with two feature datasets both of size N x p? What will be my final dataset  X and labels y?


texts: input_dim for Dense Layer after LSTM layers Keras

texts:
Do I need to specify the input_dim (which means the number of features in one row/sample) after adding the first LSTM layer for the later Dense layers?

I was trying to create an architecture with 2 LSTM layers and 1 Feed-forwarding layer with 200 cells and 1 Feed-forwarding layer with 2 cells. First LSTM layer outputs for every timestamp second LSTM layer outputs for only the last time step so I was wondering If I created that architecture correctly

model.add(LSTM(units = 200, return_sequences = True, input_shape = (WINDOW_SIZE, 9), batch_size = 206))
model.add(LSTM(units = 200, return_sequences = False))
model.add(Dense(units = 200, input_dim = 9))
model.add(Dense(units = 2, input_dim = 9))
model.add(Dense(units = 1, input_dim = 9))


texts: Confusion Matrix vs AUC Confusion with Random Forest

texts: I'm currently training a random forest classifier with 1000 trees and am receiving the following confusion matrix. I've split the training set 75/25 against the test set but am not sure why I'm seeingf such a low True Negative score?  This is a binary classification problem and my dataset contains 88% of Class 0 whereas Class 1 represents 12% of the total content.  I've tried experimenting with class_weights but I have not been able to get the True Negative score above 6%. What am I missing as it seems to pass most tests including Cross-Validation > .90 etc.


texts:
I have a list (sub_indices) containing dataframe index values. Using these index values, I need to make changes in another dataframe (main_df). Changes must be done on these index values that are in the list.

What should be the best way to solve this task? I'v tried lambda and map functions but am getting an error: "AttributeError: ("'list' object has no attribute 'apply'", 'occurred at index 0')".

Seems that lambda function cannot be called on a list.

def function_1(new_value):
    sub_indices = sub_df.index.values.tolist()
    sub_indices.apply(lambda x: function_2(x))

def function_2(df_indices):
    # Replacement that needs to be done. 'indes_value' is the next element from 'sub_indices' list.
    main_df.at[indes_value, 'column_name'] = "new value"


texts: RAM crashed for XML to DataFrame conversion function

texts:
I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) crashes. Same happens if I try it locally on Jupyter Notebook (4GB Laptop RAM). Is there a way to optimize the code?
Code
#Libraries
import pandas as pd
import xml.etree.cElementTree as ET

#Function to convert XML file to Pandas Dataframe
def xml2df(file_path):

  #Parsing XML File and obtaining root
  tree = ET.parse(file_path)
  root = tree.getroot()

  dict_list = []

  for _, elem in ET.iterparse(file_path, events=("end",)):
      if elem.tag == "row":
        dict_list.append(elem.attrib)      # PARSE ALL ATTRIBUTES
        elem.clear()

  df = pd.DataFrame(dict_list)
  return df

Part of an XML File ('Badges.xml')
<badges>
  <row Id="82946" UserId="3718" Name="Teacher" Date="2008-09-15T08:55:03.923" Class="3" TagBased="False" />
  <row Id="82947" UserId="994" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82949" UserId="3893" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82950" UserId="4591" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82951" UserId="5196" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82952" UserId="2635" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />
  <row Id="82953" UserId="1113" Name="Teacher" Date="2008-09-15T08:55:03.957" Class="3" TagBased="False" />

This conversion in needed so that I can perform furthur Data Analysis.
I have asked this on StackOverflow (Link) but the answers did not solve my query. I hope to find a solution here.


texts: Simple approach to assign topic / heading to a text

texts: I have an uncleaned text field something like Apple Juice xxx Newyork .. Store and a assigned topic Juice Centre. What are simple techniques that can be used to assign topic to unassigned texts using words/ combination of words in the text? not looking for machine learning  models at start. I have enough number of texts with assigned topics to learn some correlation.


texts: Why is orthogonal weights initialization so important for PPO?

texts:
I have implemented PPO to solve Atari environments. For the longest time I couldn't figure out why my model would not converge as fast as other open source solutions. Eventually it boiled down to this single weights initialization (in PyTorch) for the CNN:

init_ = lambda m: init(m, nn.init.orthogonal_, lambda x: nn.init.constant_(x, 0), nn.init.calculate_gain("relu"))

which is then applied to each of the layers. In other words, the neural network HAS to be initialized orthogonally. Once I added this, my model converges as expected.

Why is it that orthogonal initialization is so much superior to xavier initialization for PPO?


texts: How to make my Neural Netwok run on GPU instead of CPU

texts:
I have installed Anaconda3 and have installed latest versions of Keras and Tensorflow.

Running this command :

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

I find the Notebook is running in CPU:

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 2888773992010593937
]

This is my Nvidia version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:04_Central_Daylight_Time_2018
Cuda compilation tools, release 10.0, V10.0.130

running nvidia-smi, I'm getting this result:

I want to make the neural network to train on GPU. Please help me in switching from CPU to GPU.


texts: How to use External Data Sets in test set

texts: I have a doubt regarding usage of external datasets like gdp rate, unemployment rate... etc., in test set for time series prediction. These datasets are historical and can be used along with train set, but how can it be used along with test set ? Do we need to predict it first and then use it along test set or are there any better ways to do it ?


texts:
let's say I have year month data h1  h2 h3 .. h24
                 2004  1     1    40 42 60 ..
                  :    :     :
                 2008  12    31
I am trying to predict h1....h24 for a whole week based on historical data.
so I am trying to predict h1..h24 when I only have data h1...h24 of different year but the same month and day.

How can specify my X and Y that are used to train my predictive model using Python?

Sorry this is more organized:
let's say I have these columns and raws in my data set  year month day h1  h2 h3 .. h24
                       2004    1     1      40 42 60 ..
                        :    :      :                         :    :      :
                       2008  12        31

I am trying to predict h1....h24 for a whole week based on historical data.
so I am trying to predict h1..h24 when I only have data h1...h24 of a different year but the same month and day.

How can specify my X and Y that are used to train my predictive model using Python?


texts: Clustering, Mixed Data Set with Ordinal and Nominal Scale Data

texts:
After reading a bit how categorical data can be considered in clustering, I came to the conclusion that most of the post do not make distinction between nominal scale data e.g. colour: red, green, blue, and ordinal scale data e.g. size: small, medium, large. However, the distances between the items make sense at the ordinal scale even if they are not necessarily the same between all items.

My questions:

Can I simply convert ordinal scale data to numeric scale without
causing much trouble in clustering? I think yes, for the above reasons, but I would be pleased if you could confirm.
For nominal scale data, where the distances between the items make
no sense, would be harder to capture. The easiest way I have found,
if there are not too many items on the scale, was to break down the
scale and add a variable for each item. E.g. originally, we have
colour: red, green, blue, and we make variable colour_red,
clour_green, and colour_blue, where each of them can take a value: 0
or 1. See the post form Jordan A on
K-Means clustering for mixed numeric and categorical data.
It seems to me a valid numeric scale of type ratio as it has a
non-arbitrary zero value expressing the complete absence of
something and 1 expressing the presence of something. Do you have
experience with this in clustering? Is this a valid approach?

Otherwise I know I should use e.g. kproto (Kproto) for mixed data sets or kmodes (Kmodes) for plain, nominal data sets. Thank you for you responses.


texts: Pig is not able to read the complete data

texts:
I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR.
The operations on the data are simple (JOIN and STORE), but the data is not getting loaded completely, and the program is terminating with a java outofmemory exception.

I've tried increasing the Pig Heap size to 8192, but that hasn't worked, however my code works fine if I use only 25% of the dataset.

This is the last line of the complete log file:

  20124550 [Service Thread] DEBUG
  org.apache.pig.impl.util.SpillableMemoryManager  - memory handler call - Collection threshold init = 374341632(365568K) used =
  698749896(682372K) committed = 698875904(682496K) max =
  698875904(682496K)

  java.lang.OutOfMemoryError: Java heap space
  -XX:OnOutOfMemoryError="kill -9 %p

Here's my code for loading the dataset:

log  = LOAD s3://some-path/{20151101..20151130}/{00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23}/' USING PigStorage ('\n') AS (line:chararray);

I've specified each date individually, so that is not the issue.

Also I've tried using 35 machines in the cluster, but that hasn't helped. I could potentially add more, but that is not the type of solution I am looking for.

Can anyone can suggest better approach to coding this problem, or some other solution?


texts: Find errors if any in tensorflow computation graph

texts:
Given a computational graph in tensorflow find errors if any in the underlying code.

import tensorflow as tf
import numpy as np
input = tf.placeholder(shape=(1, 224, 224, 3), dtype=tf.float32)
output = input + 5
with tf.Session() as sess:
  network_input = np.random.randint(5,  size=(700, 350, 3))
  out = sess.run(output, feed_dict = {input : network_input})

The code is throwing error as the shape of the placeholder and the input given is not matching.
So for fixing it I am changing the shape of the place holder.
Is it the correct approach or the input given should be reshaped to the placeholder shape?


texts: Calculate a Rank function from Regression features

texts:
I am using 3 features (x1, x2, x3) for regression. Some of my features are continuous some are categorical.

My dependent variable are lets number of bookings. And I can predict the number of bookings.
By obtaining how important each feature was in regression as follows (i.e. feature importance)

x1 --> 0.1
x2 --> 0.5
x3 --> 0.7

It is clear that feature 3 (x3) contributes the most, x2 the second and x1 the least in classification.

I also performed correlation analysis to check if my features are positively or negatively correlated with the target (y) as follows.

x1 --> positively correlated
x2 --> positively correlated
x3 --> negatively correlated

I am wondering if it is possible to convert my regression features into a ranking function using feature importance and correlation.

For instance, my suggestion looks as follows.

ranking_score = 0.1*x1 + 0.5*x2 + 0.7*(1/x3)

The reason for using (1/x3) in the above equation is because it is negatively correlated with the target (y). Please let me know if my ranking_score equation is statistically correct? If not, please let me know your suggestions.

EDIT: Why ranking is important to me?

I am happy to provide more details if needed.


texts: I'm trying to create a object detection model to detect different type of milk. What is the best approach to achieve the result in the picture below? As you can see in the picture, this model did pretty well on locating and detecting different brand of milks. I've tried Faster R-CNN but they mix up products with similar shape really bad(not only on the classifier but also on the religion proposal net). I labeled my training images in the same way as in the picture below. Any help will be appreciated!


texts:
I came across this video lecture https://www.youtube.com/watch?v=wjILv3-UGM8
on k fold cross validation (CV). The algorithm given in the video lecture is presented below:

for k = 1:5
train on all except k
get model $M_{\tilde{k}}$
calculate accuracy on $k$ as $A_k$
end
Calculate final cross validation accuracy: $A =
> \frac{1}{5}\sum_{k=1}^5 A_k$

This is quite clear to me. Here  $M$ is I guess just a single type of ML algorithm. However at time stamp 6:35 the presenter raises the question that what do we do with all the 5 different models that were built? According to him, we either combine all the models and make decision based on that or take the best model out of the 5. Is this statement true?
In many sites including here (https://stats.stackexchange.com/questions/310953/doubt-about-k-fold-crossvalidation?noredirect=1&lq=1 ;  https://stats.stackexchange.com/questions/11602/training-on-the-full-dataset-after-cross-validation and https://stats.stackexchange.com/questions/11602/training-on-the-full-dataset-after-cross-validation) and research papers I have understood that:
-- for doing model training using k fold CV, we re-train on the entire dataset after the end of the CV loop and that is the final model.
-- We do not select any model from inside the CV loop if the idea of doing CV training is to check the accuracy of the ML algorithm on the entire dataset.
-- However, if we have multiple ML algorithms say random forest, neural network, SVM inside the CV loop then we select the algorithm with the highest accuracy.
-- Another technique, nested cross-validation is used for hyperparameter tuning.
Is my understanding correct?


texts:
Does anyone know how the what the kind of graph in this image is called?

Each color represents a class and the height of the color, in a particular instant, represents the ratio of elements that belong to that class.

How can I produce such a graph, as an example, in R or Python?


texts: Where can I find a good machine learning research team /university Lab for a PhD in CS?

texts:
I am aiming to do a PhD in Machine Learning and in Germany since I have a Masters in ML already I wanted to know what are the best options to aim for ?

Thank you


texts: Recommender system that connect users with each other , should I go for content based or collaborative filtering?

texts:
I am trying to build a system where user come on the platform and he chooses a topic(predefined few topics) and then we connect him with any random online user who chooses the same topic. Then they can do conversation.

Now, I am trying to connect them smartly based on user's historical data (users with whom he had match earlier along with time duration of their conversation, and raing after the conversation etc). and his basic profile data.

How can I use collaborative filtering here, because I don't have any product here and their are very few users available online(10-15) at any time so I have to connect any one of them.

Thanks in advance!