texts: Tensorflow Model fitting ValueError: Layer sequential expects 1 inputs, but it received 520 input tensors

texts:
I am trying to train a model using Tensorflow. I am reading a huge csv file using tf.data.experimental.make_csv_dataset
Here is my code:
Imports:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing

LABEL_COLUMN = 'venda_qtde'

Reading a csv into a tf.data.Dataset:
def get_dataset(file_path, **kwargs):
  dataset = tf.data.experimental.make_csv_dataset(
      file_path,
      batch_size=4096,
      na_value="?",
      label_name=LABEL_COLUMN,
      num_epochs=1,
      ignore_errors=False,
      shuffle=False,
      **kwargs)
  return dataset

Buiding a model instance:
def build_model():
  model = None
  model = keras.Sequential([
    layers.Dense(520, activation='relu'),
    layers.Dense(520, activation='relu'),
    layers.Dense(520, activation='relu'),
    layers.Dense(1)
  ])

  model.compile(loss='mean_squared_error',
                optimizer='adam',
                metrics=['mae'])
  return model

Executing the functions:
ds_treino = get_dataset('data/processed/curva_a/curva_a_train.csv')
nn_model = build_model()
nn_model.fit(ds_treino, epochs=10)

But when the fit function is called, I get the error:
ValueError: in user code:

    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:747 train_step
        y_pred = self(x, training=True)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:975 __call__
        input_spec.assert_input_compatibility(self.input_spec, inputs,
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:155 assert_input_compatibility
        raise ValueError('Layer ' + layer_name + ' expects ' +

    ValueError: Layer sequential expects 1 inputs, but it received 520 input tensors. Inputs received: ...

My dataset has 519 features and 1 label and about 17M lines.
Can anyone help me what I am doing wrong?


texts: evaluation metrics for multiple values per session

texts:
I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based on execution delay .

The number of times foo() is called per user session is variable but will not exceed 10000.
Say delays values are:

Algo1: [ [12, 30, 20, 40, 24, 280] , [13, 14, 15, 100], [20, 40] ]
Algo2: [ [1, 10, 5, 4, 150, 20] , [14, 10, 20], [21, 33, 41, 79] ]

  My question is whats the best metric to pick the winner ?

possible options

average from each session, and then evaluate cdf
median from each session and then evaluate cdf
anything else ?


texts: How to Normalize & Scale a Single Data Point

texts:
I do understand the concept of normalizing & scaling the training/test data; it does help with the converging of the cost function. It is a great helper for many of the machine learning algorithms.

I train and validate my model with the normalized data (MinMaxScaler) and save my model.
A new input data comes in and I want to use my saved model to make predictions.
I no longer have access to the training/test data at this point. All I have is the new single row of input data.
How will I normalize this single vector of input data so that it can be fed to my model?

The only normalization that I can think of simply linear transformation of data (e.g. simply map values from [40, 70] range to [-1, 1]). I need a normalization technique that doesn't depend on the full range of data.

Thanks!


texts: Visualization of how K means clustering is used to select K anchor boxes?

texts:
I was having trouble understanding how K means clustering is used to select K anchor boxes.

We have ground-truth boxes and we run K means clustering on them using IOU as a metric. There is no good graphical explanation on how IOU is used as a metric and how centroids (anchor boxes) shift with IOU.

If anyone could provide me a visualization of How K means clustering works in the context of anchor boxes that would be a huge help.


texts: How to deal with memory insufficient read by pandas in python

texts:
I use pandas.read_csv to read a huge file for machine learning, but I got memory error.

Someone recommend me to set arg chunksize but I need sort, random access...etc. So I need to load whole data into memory or use another way.

Some ways I think it's possible is Hadoop. Another one is incremental training, but it's like reading chunksize in read_csv

Or other software/library/ways can I use?


texts: Why do we need 2 matrices for word2vec or GloVe

texts:
Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix factorization of the ppmi of the co-occurrences of the words in the corpus.

Still, I can't understand why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V ? Is it a problem with the gradient descent or is there another reason ?

Someone told me it might be because the embeddings u and v of one word should be far enough to represent the fact that a word rarely appears in its own context. But it is not clear to me.


texts: How to classify movement data (time series) in real time

texts:
I have some movement data sampled over a time series. I am trying to classify the movements in real time as either smooth or shaky. For example, as the movement is smooth it is classified as smooth until it suddenly becomes shaky and is then classified as so.

I am having some serious trouble doing this. Aside from using an RNN and feeding it subsets of the time series data to classify (which I do not have enough data to train), does anybody know of any time series classification algorithms? I plan on feeding subsets of the time series into the algorithm with a single label for training and prediction.

Any help is greatly appreciated.


texts:
I have been working on developing a system "Converting Natural Language to SQL Query".

I have read the answers from the similar questions, but was not able to get the information that I was looking for.

Below is the flowchart for such system which I have got from An Algorithm to Transform Natural Language into SQL Queries for Relational Databases by Garima Singh, Arun Solanki

I have understood till part of speech tagging step. But how do I approach the remaining steps.

Do I need to train all the possible SQL queries?
Or, once part of speech tagging is done, I have to play with the words and form a SQL query?

Edit: I have successfully implemented the from step "user query" to "Part of speech tagging".

Thank you.


texts: How to retrieve images from a url in a pandas dataframe and store them as PIL object in a new column

texts:
I'm trying to store as a PIL object in a new column of a dataframe pictures that are located in a column of the same dataframe in the form of URL's.

I've tried the following code:

import pandas as pd
from PIL import Image
import requests
from io import BytesIO

pictures = [None] * 2

df = pd.DataFrame({'project_id':["1", "2"],
                    'image_url':['http://www.personal.psu.edu/dqc5255/gl-29.jpg',
                                'https://www.iprotego.com/wp-content/uploads/google.jpg']})

df.insert(2, "pictures", pictures, True)

for i in range(2):
    r = requests.get(df.iloc[i,1])
    df.iloc[i,2] = Image.open(BytesIO(r.content))

df

I expected to get a dataframe with this structure but including both training examples:

    project_id                  image_url                                  pictures
0       1    http://www.personal.psu.edu/dqc5255/gl-29.jpg <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=400x300 at 0x116EF9AC8>

But instead got the following error:

OSError: cannot identify image file <_io.BytesIO object at 0x116ec2f10>
```


texts:
I‘m looking for a minimal applied example for the implementation of a (one shot) Siamese Network, preferably in Keras.

I‘m well aware of the various data science online pages and the respective examples and exercises that can be found there. However, so far I did not found an instructive source there.

I would be thankful if someone could point me to some github source or if someone could share some code or other sources, which provide a sound example on how to implement such an architecture in Python or in R.


texts: Is there any benefits from using Doc2vec for word embedding ( replacing word2vec ) ? in other hand if I train word2vec and doc2vec with the same dataset will I have the same word vectors ?


texts: What should you do with attributes that predictive in an interaction?

texts: I am trying to predict results of football games. Some of our attributes only give meaning for a prediction only when they are considered in interaction with another attribute. To illustrate, a team with an average possession of 48.6 has much higher possession than some teams and much lower than others. Should we modify the features in such a way that we would use the difference between the two teams, instead of two values? On the other hand some algorithms might be good at finding this interactions?


texts: I have a column named GarageYrBlt which just lists the year the garage of that house was built.  I have one nan value for this column.  Does it make sense to fill it with the columns median (This was a common approach for the missing age values in the Titanic competition)?


texts: Need ideas on how to find the best email send frequency go get maximum desired action

texts:
I have got data which contains email id total times email has been send to him (frequerny) and desired action taken (read, clicked etc), non desired action taken(unsubscribed, etc), also count of no action taken which will be y= a-b-c.

Now I need to find a optimised frequency where

I expect to get best frequency of desired response or optimum frequency to send mail
Number of mails he should send in day hour or whatever. What kind of algorithm or whatever Is out there I can use. Please suggest. Is there any optimisation algo in ML or solution to such problems available.


texts: What types of features are used in a large-scale click-through rate prediction problem?

texts:
Something that I often see in papers (example) about large-scale learning is that click-through rate (CTR) problems can have up to a billion of features for each example. In this Google paper the authors mention:

  The features used in our system are drawn from a variety of sources,
  including the query, the text of the ad creative, and various
  ad-related metadata.

I can imagine a few thousands of features coming from this type of source, I guess through some form of feature hashing.

My question is: how does one get to a billion features? How do companies translate user behavior into features in order to reach that scale of features?


texts:
Can anyone tell about different Techniques/algorithms of Random forest?
I know, Random Forest is itself an algorithm/model, but I'm looking for another version of it as we have in decision trees.
List of Algorithms based on Random Forest?
Thanks


texts: Strategies for handling unlabeled data which is slightly different from the labeled data

texts:
Suppose you have a dataset with the following properties:

The number of samples is fairly large (~100K samples)
There are ~150 contextual features and 1 feature consisting of a text-string (which can, of course, be split into any number of features depending on the pre-processing of the text). It is expected that the text-string will have really great predictive power
Samples are divided into 3 categories (prior to you receiving the data) based on a few of the contextual features with category A containing ~5% of the samples, category B containing ~20%, and category C containing the remaining 75%
Category A is entirely labeled, category B is partly labeled (with only a small proportion being unlabeled), and category C is entirely unlabeled
The features used to categorize the samples are likely to influence the probability of a sample belonging to class 0 or class 1.
The samples are not completely different between categories (that is to say, we're not talking cats versus dogs). E.g.: Two very similar samples might end up in different categories based on very small differences on a numerical feature with a large range

The purpose is to build a classifier which will correctly classify the samples. That might look like a semi-supervised learning-problem, but I am worried about the structural differences between the categories. Hence my question: Which strategies could be employed to build a classifier performing well on all of the samples?

Of course I could just be conservative and only deal with the labeled data, but there is great value in also being able to predict the unlabeled data (e.g. the 75% of the data in category C). That's why I'll try to pick your brains for creative solutions!


texts: What exactly is A dataset? is a database contain more than 2 tables counted as 1 or 2 datasets?

texts:
A data set (or dataset) is a collection of data. Most commonly a data set corresponds to the contents of a single database table, or a single statistical data matrix, where every column of the table represents a particular variable, and each row corresponds to a given member of the data set in question.

It seems that the iris data set could be counted as 1 dataset.

Is a database that contains more than 2 tables counted as 1 or 2 datasets?


texts: Trying to return more than just the top result from sklearn NearestNeighbors

texts:
I'm trying to compare a list of names (duplicated into a clean file and a messy file).  I then compare the files against each other.  My problem is that it returns only the top 1 result for each, which is itself (the identical record in each file).  What I am trying to capture is the second result, which would be the closest match, not being itself.

names =  pd.read_csv('C:/Temp/messynames.txt', sep='\t')
org_names = names['VariationName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
tf_idf_matrix = vectorizer.fit_transform(org_names)

clean_org_names = pd.read_csv('C:/Temp/cleannames.txt', sep='\t')
org_name_clean = clean_org_names['StandardName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams, lowercase=False)
tfidf = vectorizer.fit_transform(org_name_clean)

nbrs = NearestNeighbors(n_neighbors=3, n_jobs=-1).fit(tfidf)
unique_org = set(names['VariationName'].values)

def getNearestN(query):
  queryTFIDF_ = vectorizer.transform(query)
  distances, indices = nbrs.kneighbors(queryTFIDF_)
  return distances, indices
distances, indices = getNearestN(unique_org)

unique_org = list(unique_org) #need to convert back to a list

matches = []
for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
matches.to_csv('C:/Temp/matchednames.txt', sep='\t', encoding='utf-8', index=False, quoting=3)

For a file with the following four names:

NOKIA
NOKIAA
NOKIA LMD
NOKIA LTD

The results looks like this:

Match confidence                         Matched name Original name
0                                 0.0    NOKIA LMD     NOKIA LMD
1                                 0.0       NOKIAA        NOKIAA
2                                 0.0        NOKIA         NOKIA
3                                 0.0    NOKIA LTD     NOKIA LTD

I'm trying to get to something more like:

Match confidence                         Matched name Original name
0                                 0.1    NOKIA LTD     NOKIA LMD
1                                 0.1        NOKIA        NOKIAA


texts: If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

texts:
Let's say
A and B are correlated
A and C are correlated
B and C is uncorrelated
How is it possible for B and C to be uncorrelated when they are both correlated to A?


texts: How to define a custom performance metric in Keras?

texts:
I tried to define a custom metric fuction (F1-Score) in Keras (Tensorflow backend) according to the following:

def f1_score(tags, predicted):

    tags = set(tags)
    predicted = set(predicted)

    tp = len(tags & predicted)
    fp = len(predicted) - tp
    fn = len(tags) - tp

    if tp>0:
        precision=float(tp)/(tp+fp)
        recall=float(tp)/(tp+fn)
        return 2*((precision*recall)/(precision+recall))
    else:
        return 0

So far, so good, but when I try to apply it in model compilation:

model1.compile(loss="binary_crossentropy", optimizer=Adam(), metrics=[f1_score])

it gives error:

TypeError                                 Traceback (most recent call last)
<ipython-input-85-4eca4def003f> in <module>()
      5 model1.add(Dense(output_dim=10, activation="sigmoid"))
      6
----> 7 model1.compile(loss="binary_crossentropy", optimizer=Adam(), metrics=[f1_score])
      8
      9 h=model1.fit(X_train, Y_train, batch_size=500, nb_epoch=5, verbose=True, validation_split=0.1)

/home/buda/anaconda2/lib/python2.7/site-packages/keras/models.pyc in compile(self, optimizer, loss, metrics, sample_weight_mode, **kwargs)
    522                            metrics=metrics,
    523                            sample_weight_mode=sample_weight_mode,
--> 524                            **kwargs)
    525         self.optimizer = self.model.optimizer
    526         self.loss = self.model.loss

/home/buda/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, **kwargs)
    664                 else:
    665                     metric_fn = metrics_module.get(metric)
--> 666                     self.metrics_tensors.append(metric_fn(y_true, y_pred))
    667                     if len(self.output_names) == 1:
    668                         self.metrics_names.append(metric_fn.__name__)

<ipython-input-84-b8a5752b6d55> in f1_score(tags, predicted)
      4     #tf.convert_to_tensor(img.eval())
      5
----> 6     tags = set(tags)
      7     predicted = set(predicted)
      8

/home/buda/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in __iter__(self)
    493       TypeError: when invoked.
    494     """
--> 495     raise TypeError("'Tensor' object is not iterable.")
    496
    497   def __bool__(self):

TypeError: 'Tensor' object is not iterable.

What is the problem here? The fact that my f1_score function inputs are not Tensorflow arrays? If so, where/how can I convert them correctly?


texts: Why is predict_generator is returning an empty array?

texts:
I am trying to print the predicted labels of my test data but the predict_generator() function is returning an empty array.

My Model:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical')

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Test'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical'
    )

filenames = validation_generator.filenames
nb_samples = len(filenames)

model = Sequential()
model.add(Flatten(input_shape = (28,28,3)))
model.add(Dense(128,activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'],)

model.fit_generator(training_generator,epochs=20,validation_data = validation_generator)
predict = model.predict_generator(validation_generator,steps = nb_samples)

print(predict)

The output :

Found 1748 images belonging to 3 classes.
Found 0 images belonging to 0 classes.
Epoch 1/20
55/55 [==============================] - 4s 64ms/step - loss: 0.9951 - accuracy: 0.4908
Epoch 2/20
55/55 [==============================] - 4s 73ms/step - loss: 0.7280 - accuracy: 0.7248
Epoch 3/20
55/55 [==============================] - 6s 109ms/step - loss: 0.5783 - accuracy: 0.7895
Epoch 4/20
55/55 [==============================] - 5s 85ms/step - loss: 0.4649 - accuracy: 0.8387
Epoch 5/20
55/55 [==============================] - 4s 71ms/step - loss: 0.3955 - accuracy: 0.8707
Epoch 6/20
55/55 [==============================] - 4s 74ms/step - loss: 0.3595 - accuracy: 0.8804
Epoch 7/20
55/55 [==============================] - 4s 70ms/step - loss: 0.3247 - accuracy: 0.8919
Epoch 8/20
55/55 [==============================] - 4s 66ms/step - loss: 0.3435 - accuracy: 0.8827
Epoch 9/20
55/55 [==============================] - 4s 65ms/step - loss: 0.2740 - accuracy: 0.9102
Epoch 10/20
55/55 [==============================] - 3s 60ms/step - loss: 0.2141 - accuracy: 0.9336
Epoch 11/20
55/55 [==============================] - 3s 61ms/step - loss: 0.1836 - accuracy: 0.9462
Epoch 12/20
55/55 [==============================] - 3s 63ms/step - loss: 0.1722 - accuracy: 0.9416
Epoch 13/20
55/55 [==============================] - 3s 62ms/step - loss: 0.1788 - accuracy: 0.9394
Epoch 14/20
55/55 [==============================] - 3s 63ms/step - loss: 0.1331 - accuracy: 0.9571
Epoch 15/20
55/55 [==============================] - 4s 68ms/step - loss: 0.1343 - accuracy: 0.9537
Epoch 16/20
55/55 [==============================] - 4s 65ms/step - loss: 0.1033 - accuracy: 0.9680
Epoch 17/20
55/55 [==============================] - 3s 62ms/step - loss: 0.1001 - accuracy: 0.9651
Epoch 18/20
55/55 [==============================] - 3s 62ms/step - loss: 0.1209 - accuracy: 0.9565
Epoch 19/20
55/55 [==============================] - 3s 61ms/step - loss: 0.1187 - accuracy: 0.9559
Epoch 20/20
55/55 [==============================] - 3s 63ms/step - loss: 0.0834 - accuracy: 0.9737
[]


texts: Are there any other data repositories like UCI and mlData, for biological data?? I want to know about mostly biological data set.


texts: Margin of error for imbalanced discrete set

texts:
I'm evaluating the performance of a classifier regarding its false negatives. The classifier performed over 9090 samples, from which 9000 were labeled as negative. I randomly chose 800 samples (out of the 9000) and found that 4 were wrong. This means a 0.5% false negative rate. I also know that the precision (in all the 90 samples labeled positive) is 10%. I want to answer questions like:

What's the probability that the false negative rate in all the 9000 labeled negative samples is greater than 2%?
What's the margin of error for a 95% confidence?


texts: How to get DATE column from dataframe in pandas without using CSV


texts:
While studying Gaussian mixture models and the expectation maximization algorithm, I also came across a number of studies that used 'Discriminative training' for the models that addressed some limitations of the EM algorithm. In particular, the use of maximum mutual information is quite common.

I understand the basic idea behind maximizing mutual information; however, I am unable to find any source that gives the equations involved in estimating the means, variances, mixing parameters and the predicted labels.

Can you please state them along with an explanation?


texts: Forecasting/predicting techniques for qualitative data?

texts:
I have a food alert dataset composed of nominal qualitative variables, such as type of alert, country of origin, action taken, etc. as well as the date on which the alert was recorded.

What techniques are there to predict what type of alert is most likely in a time window when there are only qualitative variables in our dataset?
Or given a variable (for example, country of origin = France) predict the rest of the variables?

I have used machine learning algorithms before but not with datasets composed of nominal qualitative variables.

On the other hand, what are the most appropriate techniques for clustering and classification this kind of datasets?

I know it's a broad question, so I expect broad answers to guide me in a general way

I have about 50000 registered alerts. In the example of the image not all the variables are shown


texts:
Excuse if this has been answered before.

I need to extract features and parse from a piece of text and run some analysis.
For e.g. "Plot the past 5-year sales of Apple" should give me the following

Information:

Company - Apple
Item - Sales
Period - past 5 years
Action - Plot

What deep learning techniques / algorithms should I be looking to use?

Any pointers are highly appreciated.


texts: logistic regression coefs to excel formulation

texts:
I have done a Logistic regression in r with 3 variables involving 1 interation:

lm(predicthead~vol+weekday*head, mydata)

weekday is a factor - 7 days of the week.

The coef values are :

    Estimate
(Intercept) -14.81119721
Vol 0.118819705
weekday2    -24.30795615
weekday3    -4.634180612
weekday4    -21.64229775
weekday5    0.607492058
weekday6    42.80131468
weekday7    19.16098601
heads   0.059934818
weekday2:heads  0.026481088
weekday3:heads  0.006665015
weekday4:heads  0.022825105
weekday5:heads  0.005045576
weekday6:heads  -0.041724212
weekday7:heads  -0.019254475

From my Understanding below is what I do in excel:

intercept +
(coef of vol * vol)+
(coef of corresponding week days) + #there no is value for weekday 1
(coef * head)+
(coef of corresponding weekday:head * head) #no value for weekday1 so it will be zero.

The sum of the above should be equal to predict() value - but I am not able to match it.
Could you please explain where I have done wrong in calculating predicthead value using excel?


texts: Trying to understand the result provided by np.linalg.norm function in numpy (normalisation)

texts:
I'm new to data science with a moderate math background. I'm playing around with numpy and can across the following:

So after reading np.linalg.norm, to my understanding it computes the 2-norm of the matrix. Wanting to see if I understood properly, I decided to compute it by hand using the 2 norm formula I found here:

Following computing the dot product, the characteristic equation, applying the formula for quadratic equation and taking square root of the max, I end up with a different result, namely:

So here is my question. What went wrong? Did I use the right formula? Also I couldn't find a conclusive way to get the 5. The doc he doc that says:

  If this is set to True, the axes which are normed over are left in the result as dimensions with size one. With this option the result will broadcast correctly against the original x.

But I can't wrap my head around it. What does it represent and how you compute it?

I hope the formatting and the question are clear enough.


texts: Automatic Video Editing via Neural Networks; Identify the best start and end frame to trim a video?

texts:
I have an abundance of videos from high speed cameras of golf swings. With this data, my ultimate goal is to apply to to these videos an already defined human pose estimation model.
However, a clear bottle neck in this pipeline is editing these videos to only contain the behavior in question. Each video is about 3 minutes long, but only have about 5 seconds of the aforementioned behavior (that is, these videos are mostly just people standing around). I can do this in a video editor, which is painstaking. Unfortunately, the videos are auto-logged and don't offer a chance to take more focused videos.
What I would like to do is to see if there is a way to train two models to identify appropriate starting and ending frames respectively, apply these models to new videos, and then use their predictions in order to automatically trim the videos. I would like to set up this pipeline in python, but in this stage am open to suggestions in other languages.
My feelings is that this is related to an object detection problem, but even with this I am a newbie and am not sure where to begin. Thanks in advance for any suggestions.


texts:
I wanted to plot a bar graph which shows covid cases reported in different regions in different regions of USA.
So here is my code:
import pandas as pd
import matplotlib.pyplot as plt
datainput = pd.read_csv("MD_COVID-19.csv")
fig = plt.figure()
ax = fig.add_axes([0,0,1,1])
rgn1=list(datainput['Allegany'])
rgn2=list(datainput['Anne_Arundel'])
rgn3=list(datainput['Baltimore'])
rgn4=list(datainput['Baltimore_City'])
rgn5=list(datainput['Calvert'])
rgn6=list(datainput['Caroline'])
rgn7=list(datainput['Washington'])
region=['Allegany','Anne_Arundel','Baltimore','Baltimore_City','Calvert','Caroline','Washington']
rgn1s=0
rgn2s=0
rgn3s=0
rgn4s=0
rgn5s=0
rgn6s=0
rgn7s=0
for item in rgn1:
 rgn1s +=item
for item in rgn2:
 rgn2s +=item
for item in rgn3:
 rgn3s +=item
for item in rgn4:
 rgn4s +=item
for item in rgn5:
 rgn5s +=item
for item in rgn6:
 rgn6s +=item
for item in rgn7:
 rgn7s +=item
tc=[]
tc.append(rgn1s)
tc.append(rgn2s)
tc.append(rgn3s)
tc.append(rgn4s)
tc.append(rgn5s)
tc.append(rgn6s)
tc.append(rgn7s)
ax.bar(region,tc)
plt.ylabel("Number of Cases")
plt.show()

My bar graph is shown as above. Why are only three bar graphs coming?
Where are the other four? Also how can I draw pie chart?


texts: How differential semi-gradient Sarsa updates estimated average reward?

texts:
I cannot understand the way how algorithm Differential Semi-gradient Sarsa updates its estimated average reward $\bar{R}$. The algorithm I am looking at is from Sutton's text book Reinforcement Learning:An Introduction , section 10.3.

Why does not update $\bar{R}$ using the reward $R$ got at current step like $\bar{R} = (1-\beta)\bar{R}+\beta*R$ ? Since based on definition, $\bar{R}$ is the estimated average reward. I cannot understand the reason why the updating is like this: $\bar{R} = \bar{R}+\beta*\delta$, where $\delta$ is just the TD error. Why using TD error to update the average reward?

The following figure shows the algorithm.


texts: What is the difference between the second layer of a recurrent hamming network and a simple maximum search?

texts:
In a Hamming network, the second Layer is a recurrent/competitive layer, in which its neurons are initialized by the outputs of first layer see chapter 3-10 in Neural Network Design.  The output $a$ of the second layer is calculated via the Weight-Matrix $W$ and the positive linear transfer function:
$$a_{t+1} = \mathbb{poslin}(Wa_{t})$$
For a network with 3 Neurons, the Weights can be chosen like
$$W=\left(\matrix{1 & -0.1 & -0.1\\-0.1 & 1 & -0.1 \\ -0.1 & -0.1 & 1}\right)$$
For an intial value of
$$a_0 = \left(\matrix{2.7 \\ -0.4 \\ 2.8}\right)$$
this will converge to (already rounded):
$$a=\left(\matrix{0 \\ 0 \\ 0.676}\right)$$
Neuron 0 and 1 have lost, Neuron 2 has won.
How is this different to a simple maximum search? - By which I mean iterating over $a_0$, and picking the neuron with the maximum value.


texts: does index of my data which is of type "Date time index" plays a part in reggression?

texts: I'm new to data science and I'm working on a regression problem. My question is the index of my data which is of type "Date time index"  plays a part in regression? I mean is it Okay if i drop the index ?


texts: How to deal with the difference in the range of Dependent variable in Train and Test data

texts:
I am training a XGBoost model for predicting number of applications and the minimum number of applications in training data is 40 and the maximum number of applications is 2000, while in test set minimum number of applications in training data is 400 and the maximum number of applications is 800. Hence the model is getting confused while predicting the value.
Any suggestions on how should I solve this issue?


texts: How do I convert my dataframe column into vectors using word2vec?

texts:
I have the column of Categories of vary length in my DataFrame as shown below :
Categories

[Restaurants , Drinks]
[Restaurants , Drinks , Rooftop]
[Dinner]
.
.
.

How do I convert this column into word2vec embedding ?


texts: I have an R script that generates a report based on the current contents of a database. This database is constantly in flux with records being added/deleted many times each day. How can I ask my computer to run this every night at 4 am so that I have an up to date report waiting for me in the morning? Or perhaps I want it to re-run once a certain number of new records have been added to the database. How might I go about automating this? I should mention I'm on Windows, but I could easily put this script on my Linux machine if that would simplify the process.


texts: Data mining: Clique based clustering to make comparison in social network analysis

texts:
I am a very beginner in data mining. I want to work on Clique based clustering method. I want to make a comparison between various datasets for social network analysis or community detection of social network analysis. Now I need more than 3 datasets and source code (Python code) to make the comparison in terms of social network analysis. The data set can be older or new that would not be any problem. But I want to work on at least 3 datasets from different domains and sizes to make a better comparison.

Please may help me to give enough instruction.

Thanks a lot.


texts: RNN LSTM input conflict due to generator

texts:
I am trying to do text classification with LSTM RNN on 255 length padded sequences.
My classification data looks like this
1, 'sequence 1'

The first column is the class label for the sequence. Since I am using custom word embedding (100 dims) I have created a custom data generator and used fit_generator api.
My data is converted into following format after preprocessing

X -> (batch_size, time_steps=255, 100) # 100 dim word embeddings
Y -> (batch_size, 2) # 2 for binary class labels

Following is what I have tried with custom generator
def data_generator(file):
    while True:
        with open(file, 'r') as f:
            reader = csv.reader(f)
            for row in reader:
                # Get embedded line encodes each sequence to its constituent word vectors
                # creates a 2D np array of size (max_len, 100)
                yield get_embedded_line(row[1]), to_categorical(row[0], num_classes=2)

input_layer = Input(shape=(255, 100))
lstm1 = LSTM(100, return_sequences=True)(input_layer)
dense1 = Dense(2, activation='softmax')(lstm1)
model = Model(inputs=input_layer, outputs=dense1)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(data_generator('data.csv'), steps_per_epoch=10)

The model defined above is not even compiling, it throws following error

ValueError: Error when checking input: expected input_1 to have 3 dimensions, but got array with shape (100, 1)


texts:
I have a list of files, an I use the KNN algorithm to classify these files.

dataset = pd.read_csv(file)
training_samples = get_sample_number(dataset)
X_train = dataset.iloc[:training_samples, 5:9]
y_train = dataset.iloc[:training_samples, 9]
X_test = dataset.iloc[training_samples:, 5:9]

# Feature Scaling
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.fit_transform(X_test)

# Fitting classifier to the training set
classifier = KNeighborsClassifier(n_neighbors=5, metric='minkowski', p=2)
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

Now I have my categories in my y_pred array. But I want to save the result in the file where I read the dataset. How can I link a prediction to the right row in the file (or dataset)?


texts: Why does downsampling leads classification to only predict one class?

texts:
I have a multi-class classification problem. It performs quite well but on the least represented classes it doesn't. Indeed, here is the distribution :

And here are the classification results of my former classification (I took the numbers off the labels):

.

Therefore I tried to improve classificationby downsampling the marjority classes

import os
from sklearn.utils import resample

# rebalance data
#df = resample_data(df)

if True:

    count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G = df.grade.value_counts()
    count_df = df.shape[0]
    class_dict = {"A": count_class_A,"B" :count_class_B,"C": count_class_C,"D": count_class_D,"E": count_class_E, "F": count_class_F, "G": count_class_G}
    counts = [count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G]
    median = statistics.median(counts)

    for key in class_dict:
        if class_dict[key]>median:
            print(key)
            df[df.grade == key] = df[df.grade == key].sample(int(count_df/7), replace = False)
                                             #replace=False,    # sample without replacement
                                             #n_samples=int(count_df/7),     # to match minority class
                                             #random_state=123)
# Divide the data set into training and test sets
x_train, x_test, y_train, y_test = split_data(df, APPLICANT_NUMERIC + CREDIT_NUMERIC,
                  APPLICANT_CATEGORICAL,
                  TARGET,
                  test_size = 0.2,
                  #row_limit = os.environ.get("sample"))
                  row_limit = 552160)

However it the results where catastrophic. The model accuracy and the model loss looked to have some issues :

And everything was classified in "A" on the test set.


texts: How do I figure out if subsets can be plotted in a normal Euler diagram?

texts: In some cases, it may be impossible to draw Euler diagrams with overlapping circles to represent all the overlapping subsets in the correct proportions. This type of data then requires using polygons or other figures to represent each set. When dealing with data that describes overlapping subsets, how can I figure out whether a simple Euler diagram is possible?


texts: Pre-train using sigmoid and train using ReLU?

texts:
Using RBMs to pre-train a deep net as in this example RBM, the activation function is sigmoid and makes the math much easier.

What are the implications after the initial weights are learned using sigmoid activation functions to switch to ReLU for the train phase?

I suppose that using tanh in either phase (pre-train or train) and sigmoid or ReLU in the other would cause great problems, but since ReLU and sigmoid are similar for small values, would it still render the pre-train phase useless?

The question can be more general in the sense of how much knowledge can you transfer from a neural network using sigmoid activation functions to one identical in structure, but using ReLU activation functions.


texts: weather and products sell relationship which one should be the target variable?

texts:
Let say I have this dataset

kor_cat is the categories of the product like(noodle,cookie....)
 and the kor_qty is the number of product selled

I want to know the weather has a relation with the store's sale record

but I want to know which machine learning technique is fit here
and I have the categori of the product and number of selled of that product.

How should I solve this problem with ML.


texts: System to provide guide to students about getting admissions to universities of their choice or some specific courses

texts:
I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear to ultimately land up in universities of their choice in future.

I want to this project for the process of learning and i need some starting points about how should i go about doing this .


texts:
The following paper ADADELTA: AN ADAPTIVE LEARNING RATE METHOD gives a method called Adagrad where we we have the following update rule : $$ X_{n+1} = X_n -[Lr/\sqrt{\sum_{i=0}^ng_i^2}]*g_n $$
Now I understand that this updation rule dynamically chooses the learning rate for each iteration but have the following question :

Here we see that larger gradients have smaller learning rates and smaller gradients have larger
learning rates. I don't understand why is this the property that is desired, in other words, why is this a good thing for our network


texts: Problem with basic understanding of polynomial regression

texts:
I have an understanding of simple linear regression. Clear that  results in a fitted line like this:

However, studying polynomial regression is a bit of a challenge having some questions about the process. I understand the idea that fitting a curviliear "line" could follow the data more precisely. But seeing the following Python code in scikit-learn:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

x = 2 - 3 * np.random.normal(0, 1, 20)
y = x - 2 * (x ** 2) + 0.5 * (x ** 3) + np.random.normal(-3, 3, 20)

x = x[:, np.newaxis]
y = y[:, np.newaxis]

polynomial_features= PolynomialFeatures(degree=2)
x_poly = polynomial_features.fit_transform(x)

model = LinearRegression()
model.fit(x_poly, y)
y_poly_pred = model.predict(x_poly)

suggests for me that here we "just" projecting the original features into 2nd degree new polynomial features, while still fitting the simple (straight) line on it. Still, we get the following solution with magically curved line:

Here arise my questions:

How could the straight line in figure #1 become a curved one in
figure #2 when we just introduce some new features but still fit the
original LinearRegression model? A cannot see why couldn't the same
estimator found curve in case 1 when it is able to find it in case
2? There is no hint for the estimator in the syntax in case 2 that "ok, let's apply a curve now instead of a line", right?
I read that polynomial regression is still linear, what is the exact
interpretation of this statement? "Linear" means no straight line but
any curve, is this what this want to say? If so, what is "nonlinear"?
Additionally, having read about penalty terms for polynomial regressions, I read that the
introduction of higher order features has the effect that
coefficients tend to grow along the magnitude. This is illustrated
here:

I cannot figure out here why does coefficients for a specific same feature (like x_1) increase just because there are more additional polynomials (e. g. x_2, x_3 etc) while stay low just being alone? Quite confusing.


texts: Training textblob with 16k rows of labeled data won't work (only few are working)

texts:
I've got labeled data in a csv which looks like:

title,type
Women Jacket A,Clothes
Mens Running Shoes B,Shoes
Children backpack,Bags

and a script:

from textblob.classifiers import NaiveBayesClassifier

with open('train_small.csv', 'r') as fp:
    cl = NaiveBayesClassifier(fp, format="csv")

print ( cl.classify("Womens running shoe") )

If I use train_small.csv with 3 categories and 32 rows, I get the printed result, correct and immediate.

If I use my full training set (train.csv, 16k rows, 10 categories) I won't get a result. Today, I ran my laptop for 6 hours and then I had to shut it down by pressing the power button since it was at a load of 60 and irresponsive.

System is:

Operating System: Kubuntu 19.04
Kernel Version: 5.0.0-15-generic
OS Type: 64-bit
Processors: 4 × Intel® Core™ i5-7200U CPU @ 2.50GHz
Memory: 7.7 GiB of RAM

Not great, but completely insufficient ..?
Do I need to use another library / algorithm or just different hardware?


texts: Path prediction using the historic path taken by person

texts:
Suppose I have a dataset which has the columns date, time, latitude, longitude, place name visited by a user. I am wondering whether there is a way to write any algorithm to predict the path, the user would take at a particular time.

For example, I have the last two months date, time, latitude and longitude, can we predict the path the user would take at a particular date next month. I am not sure whether this is possible. But if any one can give me a idea of algorithm or any packages in R or python, it would be helpful. I can research on it.


texts: Data cleaning: Relationships between columns

texts:
I have a training data set distributed in two files.

File 1: This contains actual classification for each X1. X1 is unique in this file. X1 has one-to-one relationship with X2, i.e. X2 is also unique. Y is binary.

| X1 | X2 | Y  |
| 1  | 4  | 0  |
| 3  | 5  | 1  |
...
| 8  | 9  | 1  |

File 2: This contains the real 'observations' of the experiment. X1 can appear multiple times.

| X1 | X3 | X4 |
| 3  | 4  | 5  |
| 3  | 1  | 2  |
...
| 1  | 4  | 8  |

Here I can combine the two tables to have a structure like below and use them as observations:

| X1 | X2 | X3 | X4 | Y |
| 3  | 5  | 4  | 5  | 1 |
| 3  | 5  | 1  | 2  | 1 |
...
| 1  | 4  | 4  | 8  | 0 |

For test data I have similar structure, just the Y column is missing in File 1.

I have multiple concerns here:

X1 and X2 has one-to-one dependency in the data, i.e. X1 = f(X2) and X2 = f(X1)
Y = f'(X1) or f'(X2)
Frequency distribution of X1,X2 and Y changes dramatically in the new joined data set.

Questions:

Does this kind of transformation of data leads to any insights?
Does regression and ensemble learning techniques are capable of capturing these internal relationships?


texts: Implementing the SVHN CNN architecture in Srivastava et al. 2014 Dropout paper

texts:
I am trying the implement the CNN architecture introduced in Srivastava et al. 2014 Dropout paper (appendix B.2), for the SVHN dataset. I implemented only the convolutional layers part, without dropout or any regularization for the moment, following this description of the mentioned paper:
"The convolutional layers have 96, 128 and 256 filters respectively.
Each convolutional layer has a 5 × 5 receptive field applied with a stride of 1 pixel. Each
max pooling layer pools 3 × 3 regions at strides of 2 pixels."

This is my code, implemented in TensorFlow 2.0 with Keras API

    from tensorflow.keras import layers, Sequential

    model = Sequential(name= "fMap_svhn_DANN")

    model.add(layers.Conv2D(filters= 96, kernel_size= 5, activation= 'relu', input_shape= (32,32,3)))
    model.add(layers.MaxPool2D(pool_size= 3, strides= 2))

    model.add(layers.Conv2D(filters=128, kernel_size= 5, activation= 'relu'))
    model.add(layers.MaxPool2D(pool_size= 3, strides= 2))

    model.add(layers.Conv2D(filters= 256, kernel_size= 5, activation= 'relu'))
    model.add(layers.MaxPool2D(pool_size= 3, strides= 2))

    model.add(layers.Flatten())

And this is the error I get"

Negative dimension size caused by subtracting 5 from 4 for 'conv2d_11/Conv2D' (op: 'Conv2D') with input shapes: [?,4,4,128], [5,5,128,256].

Any idea to help me ?


texts: Preprocessing and feature selection in group k fold

texts:
I have experimental data collected from 10 people. From each person, 100 data points were collected under condition A, and 100 data points were collected under condition B. So, in total I have 10*(100+100) = 2000 data points.
I would like to create a classification type machine learning model which can tell whether a data point from a previously unseen person belongs to condition A or B.
One way to do it, of course, training the model on 9 people, and testing on the other. Then I would normalize columns, do feature selection etc. on 9 people, and finally test the other person.
Unfortunately, the model performance highly depend on the person chosen for the test set. When I look at similar studies, they simply create 10 models, in which every person is the test set once and only once (group K fold). Then they report the average accuracy. But the papers I have read did not do column normalization and feature selection.
My question is: How can I do column normalization and feature selection on my dataset? Can I ...

normalize columns of all data (from 10 people) without looking at labels (condition A and B) ?
select features without looking at labels, for example, selecting uncorrelated columns? (because I have many features)


texts:
The question is about a wrongly chosen strategy for train/test splitting in a RandomForest model. I know choosing the test set this way gives the wrong output but I would like to understand why.

(The model looks at previous days of data and tries to predict if next day's data will be higher or lower than today, i.e. a classification problem)

I copied the train/test split code from another example and it simply sets random rows to be either train_set or test_set. (Tried to illustrate below)

Raw data is daily Close values of for example EURUSD.

I then create features based on that. Every feature looks at some previous data points and comes up with a set of features which is a row in X_test. I then train a random forest model to try to predict next day's close.

The accuracy in the test_set is very high and it increases with the increasing number of historical previous points it looks at which seems to suggest overfitting.

When I change train/test split model to have, for example, train_set: data in January-June and test_set: data in August, i.e. completely separate datasets and no possible mixing, the accuracy is a more realistic 50%.

Again, I know the train/test split it is not correct, but can someone help me understand why..?

My question is simple: Every time I want to validate a row (i.e. one prediction in test_set) I use features that looks at previous data trying to predict tomorrow's data? How come there is overfitting?


texts: Why my training and testing set are about 99% but my single prediction does wrong prediction?

texts:
I have performed fruits classification using CNN but i am paused at a point where all things are going right confusion matrix accuracy score all are correct it seems there is no overfitting but it always classifies wrong fruit. Why would this happen. Link to source code is provided below. Thank You!
Github source code link


texts: Finding the kernel in which the data will have the highest ratio of leading eigenvalue to trace

texts: I have a method which works well with the data where a ratio of a leading eigenvalue or of first few leading eigenvalues to trace is high. Is it possible to find a kernel transformation which is optimized in such a way (across possible values of hyperparameters and functional forms) that in the kernel space such ratio is at the highest possible value (and higher than in the original feature space)? Thank you.


texts: What is difference between text classification and topic models?

texts: I know the difference between clustering and classification in machine learning, but I don't understand the difference between text classification and topic modeling for documents. Can I use topic modeling over documents to identify a topic? Can I use classification methods to classify the text inside these documents?


texts: Image features (produced by VGG19) do not properly train an ANN in Keras

texts:
I've used a VGG16 network to extract features from an image dataset, creating a features dataset using the following code:

model = VGG16(weights='imagenet', include_top=False)
x = 0

for folder in os.listdir(os.fsencode('training_cnn_images/')):
    folder = str(folder)[2:-1]
    directory = os.fsencode('training_cnn_images/' + folder)
    dn = str(directory)[2:-1]
    for file in os.listdir(directory):
        fn = str(file)[2:-1]
        img = image.load_img(dn + '/' + fn, target_size=(224, 224))
        img_data = image.img_to_array(img)
        img_data = np.expand_dims(img_data, axis=0)
        img_data = preprocess_input(img_data)
        vgg16_feature = model.predict(img_data)
        with open(('training_cnn_features' + dn[19:] + '/' + fn)[:-4] + '.pkl', 'wb') as f:
            pkl.dump(vgg16_feature, f)
        x += 1
        printProgress(x, 1895, 'Generating Features:')

I simply feed an image to VGG16, and the resultant output is used as the extracted features.
Now that I have a .pkl dataset containing the features for all the images of the dataset, I wanted to create an ANN that takes these features as an input and classifies the images (technically their features) into two categories.

So, I wrote this code to do just that:

inney = [None] * 315
outey = [None] * 315

x = 0
for file in os.listdir(os.fsencode('training_cnn_features/n02085620-Chihuahua/')):
    fn = str(file)[2:-1]
    with open('training_cnn_features/n02085620-Chihuahua/' + fn, 'rb') as f:
       inney[x] = pkl.load(f).flatten()
       outey[x] = np.array([[0, 1]])
    x += 1
for file in os.listdir(os.fsencode('training_cnn_features/n02085782-Japanese_spaniel/')):
    fn = str(file)[2:-1]
    with open('training_cnn_features/n02085782-Japanese_spaniel/' + fn, 'rb') as f:
       inney[x] = pkl.load(f).flatten()
       outey[x] = np.array([[1, 0]])
    x += 1

inney = np.asarray(inney)
outey = np.asarray(outey)

model = Sequential()
model.add(Dense(1024, input_shape=(25088,), activation='relu'))
model.add(Dense(2, activation='softmax'))
model.compile(loss = "mean_squared_error", optimizer = optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0))
model.fit(inney, outey, epochs=10, batch_size=4, verbose=1)
model.save('test.h5')

I had also created a test dataset by partitioning the original dataset into a training and test set. When I attempted to test the network, the network always seems to output [0, 1] or [*some very small value*, 1]. When I try with sigmoid, it flips, i.e, it becomes [1, 0].

A similar issue arises (one output is always one or very close to it while the other is always zero or very close to it) when I use inceptionV3.

Nothing seems to work. I've attempted adding the extra layers directly to the inception or VGG16 network, and freezing the non-classifier layers, but I face the same issue over and over again. The neural network just won't classify.

I've attempted using both sigmoid and softmax, but face the same issue in both cases.

Help would be very much appreciated.
Thanks.


texts:
In the context of Machine Learning, I have seen the term Ground Truth used a lot. I have searched a lot and found the following definition in Wikipedia:

In machine learning, the term "ground truth" refers to the accuracy of the training set's classification for supervised learning techniques. This is used in statistical models to prove or disprove research hypotheses. The term "ground truthing" refers to the process of gathering the proper objective (provable) data for this test. Compare with gold standard.
Bayesian spam filtering is a common example of supervised learning. In this system, the algorithm is manually taught the differences between spam and non-spam. This depends on the ground truth of the messages used to train the algorithm – inaccuracies in the ground truth will correlate to inaccuracies in the resulting spam/non-spam verdicts.

The point is that I really can not get what it means. Is that the label used for each data object or the target function which gives a label to each data object, or maybe something else?


texts: multi items forecasting: issue with storing results

texts:
disclaimer: I am not 100% sure that this is the appropriate place to ask this question.
Here is a little bit of context about the problem.
I have a dataset containing about 1000 products timeseries (about two years of data)
From this data I am forecasting 12 months ahead, with several prediction intervals.
To predict, I made a basic model with statsmodels.tsa.statespace.exponential_smoothing.ExponentialSmoothing model.
I am facing an issue regarding how to store the data once the model produces results at the item level.
Here is a basic reproduction of what my script look like:

seasonal_profile_df.set_index('Id')
forecast_df = pd.DataFrame(seasonal_profile_df.index)

def winter_holts(i):
    fit1 = ExponentialSmoothing(new_df.iloc[:,i], trend=True, seasonal=12).fit()
    prediction_interval = fit1.get_forecast(steps=12).summary_frame(alpha=[0.10, 0.05, 0.01])
    forecast = pd.DataFrame(prediction_interval)
    return forecast

def holts(i):
    fit1 = ExponentialSmoothing(new_df.iloc[:,i], trend=True).fit()
    prediction_interval = fit1.get_forecast(steps=12).summary_frame(alpha=0.10)

    forecast = pd.DataFrame(prediction_interval)
    print("FORECAST",forecast)

    return forecast

for i in seasonal_profile_df.index:
    if seasonal_profile_df['trend'].loc[i] == "trending":
        holts(i)
    else:
        if seasonal_profile_df['seasonality'].loc[i] == "seasonal":
            winter_holts(i)

for each item, the forecasting function returns a dataframe that looks like that:
FORECAST 100221           mean    mean_se  mean_ci_lower  mean_ci_upper
2020-07-31  -4.412599  24.526896     -44.755753      35.930555
2020-08-31  -5.848380  24.526896     -46.191534      34.494775
2020-09-30  -7.284160  24.526898     -47.627317      33.058996
2020-10-31  -8.719941  24.526900     -49.063101      31.623220
2020-11-30 -10.155721  24.526903     -50.498887      30.187445
2020-12-31 -11.591502  24.526908     -51.934676      28.751672
2021-01-31 -13.027282  24.526915     -53.370468      27.315903
2021-02-28 -14.463063  24.526924     -54.806263      25.880137
2021-03-31 -15.898844  24.526935     -56.242062      24.444375
2021-04-30 -17.334624  24.526949     -57.677865      23.008617
2021-05-31 -18.770405  24.526966     -59.113674      21.572864
2021-06-30 -20.206185  24.526986     -60.549487      20.137117

the computations results are truly an issue right now because they are the basis for deeper analysis and they will end up in a posgresql DB.
I am inexperience to this and I am wondering how to deal with the output to be the most efficient possible as I will need to manipulate them later on in the script.


texts: Python - Predicting data based on multidimensional array with Keras

texts:
I've a list of data which is so called 3D array. Each of 10350 rows contains a 2D matrix with size of 150x16 (elements are float) (x_train). Corresponding training data for this huge array a linear array with size of 10350 integer which can be either 0 or 1 (y_train).

I used different methods to estimate the test_data. None of them worked with such type of array.

from matplotlib import pyplot as plt

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from sklearn.preprocessing import MinMaxScaler

min_max_scaler = MinMaxScaler()

train_data_set = np.load("files/samples/A08_Block_sample.npy")
test_data_set = np.load("files/samples/A10_BLOCK_sample.npy")

x_train = []
y_train = []

for i in range(len(train_data_set)):
   x_train.append(train_data_set[i].arr_)
   y_train.append(train_data_set[i].flag_)

   num_units = 4
   activation_function = 'sigmoid'
   optimizer = 'adam'
   loss_function = 'mean_squared_error'
   batch_size = 5
   num_epochs = 100

   regressor = Sequential()

   regressor.add(LSTM(units=num_units, activation=activation_function, input_shape=(10403, 16, 150)))

   regressor.add(Dense(units=1))

   regressor.compile(optimizer=optimizer, loss=loss_function)

   regressor.fit(x_train, y_train, batch_size=batch_size, epochs=num_epochs)

   inputs_x = []
   inputs_y = []
   for i in range(len(test_data_set)):
      inputs_x.append(test_data_set[i].arr_)
      inputs_x.append(test_data_set[i].flag_)

   predicted_price = regressor.predict(inputs_x)

The library that I'm using is Keras. I'm new at data science, any suggestion will be welcome for me.


texts: How to define a custom resampling methodology

texts:
I'm using an experimental design to test the robustness of different classification methods, and now I'm searching for the correct definition of such design.

I'm creating different subsets of the full dataset by cutting away some samples. Each subset is created independently with respect to the others. Then, I run each classification method on every subset. Finally, I estimate the accuracy of each method as how many classifications on subsets are in agreement with the classification on the full dataset. For example:

Classification-full     1    2    3    2    1    1    2

Classification-subset1  1    2         2    3    1
Classification-subset2       2    3         1    1    2
...

Accuracy                1    1    1    1  0.5    1    1

Is there a correct name to this methodology? I thought it can fall under bootstrapping but I'm not sure about this.


texts: What exactly is convergence rate referring to in machine learning?

texts:
My understanding of the term "Convergence Rate" is as follows:

Rate at which maximum/Minimum of a function is reached, so in logistic
regression rate at which gradient decent reaches global minimum.

So by convergence rate I am guessing it is measure of:

time measured from start of gradient descent until it reaches global maximum.
average number of distance our model went downhill(Do not know technical term...) for each iteration.

Can someone verify whether or not one of my guess is True if not explain what it means?


texts: Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

texts: Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?


texts:
I would like to train a GCN for protien-ligand binding affinity regression. I use GCNConv from pytorch geometric, ReLU for all activations and Dropout (0.2) after 2 dense layers each. Using ReLU for all activation functions.
While my training curves look overall at the same level , i noticed weird periodic jumps at validation stage . What might be the issue with the model or/and training process?
For me it seems that the network starts initializing again after each epoch, but it is not what my code seems like doing.


texts:
I am curious if ordinal features are treated differently from categorical features in decision tree, I am interested in both cases where target is categorical or continuous.

If there is a difference, could you anybody point to good source with explanation and any packages (R or Python) supporting it?


texts: My naive (ha!) Gaussian Naive Bayes classifier is too slow

texts:
I am trying to build a film review classifier where I determine if a given review is positive or negative (w/ Python). I'm trying to avoid any other ML libraries so that I can better understand the processes. Here is my approach and the problems that I am facing:

I mine thousands of film reviews as training sets and classify them as positive or negative.
I parse through my training set and for each class, I build an array of unique words.
For each document, I build a vector of TF-IDF values where the vector size is my number of unique words.
I use a Gaussian classifier to determine: $$P(C_i|w)=P(C_i)P(w|C)=P(C_i)*\dfrac{1}{\sqrt{2\pi}\sigma_i}e^{-(1/2)(w-\mu_i)^T\sigma_i^{-1}(w-\mu_i)}$$ where $w$ is the my document in a vector, $C_i$ is a particular class, $\mu_i$ is the mean vector and $\sigma_i$ is my covariance matrix.

This approach seems to makes sense. My problem is that my algorithm is much too slow. As an example, I have sampled over 1,500 documents and I have determined over 40,000 unique words. This mean that each of my document vectors has 40,000 entries and if I were to build a covariance matrix, it would have dimensions 40,000 by 40,000. Even I were able to generate the entirety of $\sigma_i$, but then I would have to compute the matrix product in the exponent, which will take an extraordinarily long time just to classify one document.

I have experimented with a multinomial approach, which is working well. I very curious on how to make this work more efficiently. I realise the matrix multiplication runtime can't be improved, and I was hoping for insight on how others are able to do this.

Some things I have tried:

Filtered any stop words (but this still leaves me with tens of thousands of words)
Estimated $\sigma_i$ by summing over a couple of documents.


texts: How to compare and find common values from different columns in same dataframe?

texts:
I would like to compare two columns and find common value sets in each column, then output the rows with the common values.

Let's say I have a dataframe with:

no.(col1) | Username (col2) | Referral(col3) | email(col4)

0 | john | mike | email0@email.com

1 | peter | paul | email1@email.com

2 | joan | patricia | email2@email.com

3 | mike | john | email3@email.com

The output would be "0 | john | mike | email0@email.com" and "3 | mike | john | email3@email.com" because they have the same values in col2 and col3 respectively.


texts: ValueError: The number of classes has to be greater than one; got 1 class in MultiLabel Classification problem

texts:
I am working in Python in a Multilabel Classification problem. I have a dataset with texts and around 20k unique labels. I transformed the text to word embeddings and now I use that in a ChainClassifier to predict the tags that each texts might have.

from skmultilearn.problem_transform import LabelPowerset, ClassifierChain, BinaryRelevance
from sklearn.svm import SVC

df = pd.read_csv('sample_fos.csv')
df = df.dropna()
df.fos = df.fos.str.split(',')

mlb = MultiLabelBinarizer()

data = df.join(pd.DataFrame(mlb.fit_transform(df.pop('fos')),
                          columns=mlb.classes_,
                          index=df.index))

labels = mlb.classes_

X_train, X_test, y_train, y_test = train_test_split(data[features], data[labels], test_size=0.20, shuffle=True)

clf = ClassifierChain(classifier=SVC(gamma="auto"))
clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)

I get the following error:

  File "<ipython-input-54-976064cc6433>", line 30, in <module>
    clf.fit(X_train,y_train)

  File "/lib/python3.7/site-packages/skmultilearn/problem_transform/cc.py", line 155, in fit
    X_extended), self._ensure_output_format(y_subset))

  File "/lib/python3.7/site-packages/sklearn/svm/base.py", line 147, in fit
    y = self._validate_targets(y)

  File "/lib/python3.7/site-packages/sklearn/svm/base.py", line 521, in _validate_targets
    " class" % len(cls))

ValueError: The number of classes has to be greater than one; got 1 class

In my knowledge, this means that one of the label columns has one class only (either 0 or 1)

However, running the following snippet, I get that all of my columns has 2 unique values

b = []
for i in labels:
    b.append(len(data[i].unique()))
print(min(b),max(b))
>> 2, 2


texts: Should the output of regression models, like SVR, be normalized?

texts:
I have a regression problem which I solved using SVR. Accidentally, I normalized my output along with the inputs by removing the mean and dividing by standard deviation from each feature.

Surprisingly, the Rsquare score increased by 10%.

How can one explain the impact of output normalization for svm regression?


texts:
What are the differences between SVC, NuSVC, and LinearSVC?

Please shed some light.


texts: Lets say you want to detect something of a picture made with the bird perspective (ie with a drone). Can you create a dataset with the pictures and the self pre generated probability maps as labels, so that you train the input-image on the output-map?


texts:
I have read online about ways to create HD images using Deep learning. StyleGAN is the the often quoted one. But it very expensive to train on new set of images. It takes around 14 days to 60 days as per their documentation (using only 1 GPU). This is for 1024x1024 size image.
My questions are:

Is there a better and cheaper way?
I also haven't found something like "Transfer learning" for this problem. I mean, take a trained base network and use custom images to train generation of HD images. Is there anything like that?

Thank you before hand.


texts: Performance metric in recommender systems with implicit feedback

texts:
This paper describes a technique for making recommendations when the feedback is implicit, that is, $r_{ui}$ is only a guess. The recommendation problems boils down to the following optimisation problem:
$$
\min_{x^*, y^*}\ \ \sum_{u,i}c_{ui}(p_{ui}-x_i^Ty_i)^2+\lambda \left(\sum_u\|x_u\|^2+\|y_u\|^2 \right)
$$
where $c_{ui} = 1+\alpha r_{ui}$.

I have the following concerns:

The algorithm above approximates the binary indicator function $p_{ui}$, weighted by the term $c_{ui}$. However, most people use RMSE between $r_{ui}$ and $x_i^Ty_i$ for assessing performance. I am not sure why that would make sense.
Why using RMSE at all? With implicit feedback, it would make more sense to me to use rank-based metrics. I would imagine RMSE is more suitable in machine learning competition where the objective metric is usually RMSE.


texts: Getting the following error with my LSTM in R

texts:
I am getting the following error whenever I train my model in R:

model <- keras_model_sequential()

model %>%
  layer_lstm(units=data_dim, input_shape = c(timesteps, data_dim),return_sequences=TRUE, use_bias=TRUE,dropout=0.35,recurrent_dropout=0.35) %>%
  layer_activation("linear") %>%
  layer_lstm(units=data_dim,return_sequences=TRUE, use_bias=TRUE,dropout=0.35,recurrent_dropout=0.35) %>%
  layer_activation("linear") %>%
  layer_lstm(units=5*data_dim,return_sequences=TRUE, use_bias=TRUE,dropout=0.35,recurrent_dropout=0.35) %>%
  layer_activation("linear") %>%
  layer_dense(units=out_dim,activation="linear")

optimizer <- optimizer_rmsprop(lr = 0.01)

model %>% compile(
  loss = "mean_squared_error",
  optimizer = optimizer
)

model %>% fit(
    train_in, train_out,
    batch_size = batch_size,
    epochs = 1000)

  UserWarning: Method on_batch_end() is slow compared to the batch update (1.533262). Check your callbacks.

Can somebody please explain what's going on?


texts:
I am trying to overfit to the maximum a random forest classifier using scikit-learn to make some tests.

Does somebody know what  hyperparameters I can tune to do that? Or does somebody know which other model I could apply to achieve a overfitted to the maximum a non-linear model?


texts: How to convert classification maps to bounding box

texts:
I have a fully convolutional network that has been trained to classify cats and non-cats in small images (48x48). Because it is fully convolutional, I would expect that if I run it in bigger images, the result will be a classification map with overlapping 48x48 patches where I can predict bounding boxes. However, I am not sure how to convert this map (and the bounding boxes) to the original input dimensions.
I have tried something along the lines of:
$$
x_i = (\left \lfloor{x_{c,1} * R}\right \rfloor, \left \lceil{x_{c,2} * R}\right \rceil)
$$
where $x_{c}$ represents the x-coordinates of the bounding box in the classification map and $x_i$, in the input image space. $R$ is the ratio between the sizes of the classification map and input image. This doesn't work.
Can someone please explain how to do it?
Thanks!


texts: How do I build a recommend system based on user's past purchases?

texts:
I am exploring approaches to build a model that shows personalized search results (with or without query) for a fashion eCommerce platform. For that I am first working on coming up with a bunch of products for each user and their corresponding likelihood to buy it.
I have user's purchase history i.e the list of all the products with the client has bought with brand name and the dress category that the product belongs to (shoe or top etc.)
So I want to populate client's search results with the items that the client is most likely to purchase based on what he has bought in the past. So I am trying to build a model that estimates the probability that the user is going to like a suggested item. The products are part of a larger product inventory.
Is this a content based filtering problem? Currently I am trying to create each client's vector profile based on his liking towards a brand or not? Is this the current way.


texts: How to merge personalized models together

texts:
Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude-longitude, and current date and time.

We have millions of users, but each user needs will be probably too unique for generalization. I.e., everyone's commute is so different that what you have learned from other users are probably not applicable to other users.

In conclusion, I have to build millions of models, how can I merge these models together for a better deployment process?
If not, what are the best practices to deploy millions of models?


texts: Training CNN: Understanding number data generated while training the model

texts:
I am training CNN on kaggle and my training and test datasets shapes are as follows:
(12271, 100, 100, 1) train shape
(3068, 1, 100, 100) test shape

(12271,) train target shape
(3068,) test target shape

But when I am training my network with below code, I am getting fllowing output while training, what does it mean. Does it mean  validation score is only calculated on 95 samples instead of 3068? I am not sure what does this number 95 denotes at the start of each epoch?
Why is it happening?

for i in range(len(model)):
    model[i].fit_generator(datagen.flow(x_train,y_train, batch_size=batch_size),
                                        epochs = epochs, steps_per_epoch=x_train.shape[0] // batch_size,
                                        validation_data = (x_test,y_test),
                                        callbacks=[ReduceLROnPlateau(monitor='loss', patience=3, factor=0.1)],
                                        verbose=2)
    models.append(model[i])

Epoch 1/700
95/95 - 12s - loss: 3.4509 - accuracy: 0.3898 - val_loss: 2.8512 - val_accuracy: 0.4198
Epoch 2/700
95/95 - 11s - loss: 3.2385 - accuracy: 0.4420 - val_loss: 4.5017 - val_accuracy: 0.0975
Epoch 3/700
95/95 - 12s - loss: 2.7225 - accuracy: 0.4928 - val_loss: 4.3058 - val_accuracy: 0.4654
Epoch 4/700
95/95 - 11s - loss: 2.5231 - accuracy: 0.5203 - val_loss: 3.7107 - val_accuracy: 0.4501
Epoch 5/700
95/95 - 11s - loss: 2.2600 - accuracy: 0.5358 - val_loss: 1.9277 - val_accuracy: 0.5639
Epoch 6/700
95/95 - 12s - loss: 2.0255 - accuracy: 0.5529 - val_loss: 1.7618 - val_accuracy: 0.5896
Epoch 7/700
95/95 - 11s - loss: 1.8013 - accuracy: 0.5756 - val_loss: 1.4528 - val_accuracy: 0.6040
Epoch 8/700
95/95 - 11s - loss: 1.6182 - accuracy: 0.5942 - val_loss: 1.1788 - val_accuracy: 0.6858
```


texts: How do tech-companies employ Random Forest on large data sets?

texts:
The algorithm takes quite a long time to train on large data sets with a moderate number of parameters:

https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r

https://stackoverflow.com/questions/34997134/random-forest-tuning-tree-depth-and-number-of-trees

https://stackoverflow.com/questions/31278688/how-can-you-reduce-the-default-ntree-500-parameter-passed-to-rf-from-caret

I've been trying to run it on a ~25,000 row data set with 36 predictors and it has been using 6GB of RAM for over 2 hours.

Are there instances where this algorithm is used in production or is being run daily? If so, how does one approach re-training it or optimising it for large data sets?


texts: Pre-process data images before training OneClassSVM and decrease number of features

texts:
I want to train a OneClassSVM() using sklearn, and I have a set of around 800 images in my training set.

I am using opencv to read the images and resize them to constant dimensions (960x540) and then adding them to a numpy-array. The images are RGB and thus have 3-dimensions. For that, I am reshaping the numpy array after reading all the images:

#Assume X is my numpy array which contains all the images before reshaping
#Now I reshape X
n_samples = len(X)
X = X.reshape(n_samples, 950*540*3)

As you can see, the number of features is huge (1,539,000 to be exact).

Now I try to train my model:

model = OneClassSVM(kernel='rbf', gamma=0.001)
model.fit(X)

After running my code, it crashed due to MemoryError. If I'm not mistaken this is obvious due the large number of features? So, is there a better way to pre-process the images before fitting them, or to decrease the number of features?


texts: Understanding these probabilities, and reading the pdf and cdf

texts:
UPDATE: this is a math question but as i would like to codify it, I thought the DS SE might be appropriate.

I am trying to sketch out this problem statement. Where am I going wrong since I don't see how they get

"6% chance of losing any money at all",
"41% chance of making more    than \$100M" and
"75% of earning at least \$40M".

My attempt at a solution is in the second image: Row 48 (CASES) enumerates the number of ways the sum can take a given value, while row 49 (PROB) is that number in row 48 divided by the total number of outcomes. This gives me the pdf and row 50 is the cdf. Yet I don't get the answers in the problem statement. Are they wrong or am I reading the pdf and cdf the wrong way?

The first image is the text below.


texts: What is the best way to split a sentence for a keyword extraction task?

texts:
I'm doing a keyword extraction using TF-IDF on a large number of documents. Currently, I'm splitting each sentence based on n-gram. More particularly, I'm using trigrams. However, this is not the best way to split each sentence into ints constituting keywords. For example a noun phrase like 'triple heart bypass' may not always get detected as one term.

The other alternative to chunk each sentence into its constituting elements look to be part of speech tagging and [chunking][1] in [Open NLP][2]. In this approach phrase like 'triple heart bypass' always gets extracted as a whole but the downside is in TF-IDF the frequency of extracted terms (phrases) dramatically drops.

Does anyone have any suggestion on either or these two approaches or have any other ideas to improve the quality of the keywords?


texts: Combining Latitude/Longitude position into single feature

texts: I have been playing with 2 dimensional machine learning using pandas (Trying to do something like this:  https://github.com/freeman-lab/spark-ml-streaming), and I'd like to combine Lat/Long into a single numerical feature -- ideally in a linear fashion. Is there a "best practice" to do this?


texts:
I am new to the ML/DS field. I started a project where I need to predict the age of users. I'd like to build two models, one predicting the age group (I created 6 age groups), and the other one predicting the age explicitly (as a number). I cleaned the data, and did the correlation between the possible input variables and target variables.
What I get confused about is: how many variables is too little compering to the dataset, and how many variables is too much?
Some people are suggesting that "the more the better", but I am also aware that too many features can cause overfitting...
Is there a rule of thumb? ex: "if I have a dataset of 500k it's reasonable to have 50 variables..." or is this really depending from case to case, and I should figure out variable importance to the model, once it is done?
Please let me know if something needs to be clarified, I'd really like if someone could help me with their experience and/or direct me to an article/paper that deals with this subject.


texts: why does adding an LDA document vector with a word2vec word vector work well in LDA2vec?

texts:
In LDA the document weight vector represents the "weights" of each topic in the document. I think it's also valid to say, each row in the document vector corresponds to a word in the document, the weight value being indicative of how relevant each word is to the topics being discussed in the document.

where dj is some document vector in the form

from my understanding, LDA2vec tries to minimize the cost function by  shifting the p_jk weights of the document vector such that the sum w_j dot c_j for j = 0...n is minimal, which makes sense. I just don't understand the relationship between LDA space and the space of word vectors, because from the way it's being described it seems like the rows of the word vectors carry the same meaning as the rows of the document vectors in LDA, and I don't understand why that would be. If anyone can clear the confusion, that would be greatly appreciated, thanks in advance.

Update:

After further thought, the only way this would make sense to me is if the word, topic, and document vectors weights were being jointly trained and therefore belonging in a common space. Not sure if this is the case though.


texts: what is meant by minimizing and maximizing in GANs?

texts:
It is a subtle change that involves the generator maximizing the log of the discriminator probabilities for generated images instead of minimizing the log of the inverted discriminator probabilities for generated images.

This is a very tricky and confusing line to understand GANs loss functions, it is taken from machine learning mastery blog. My question is about the terms minimizing the log of the inverted discriminator probabilities, isn't the subtraction of probabilities from 1 give you same thing (referring to the formula of this part of loss function). please clarify it with some kind of analogy or example of some sort.


texts:
I was researching on evaluation metrics to understand the performance of unsupervised anomaly detection algorithms and I came across this paper
The author suggests that EM and MV based numerical criteria are able to recover which algorithm is better than the other on some dataset (with potentially large dimensionality), without using labels.
Could someone explain the concept of EM?


texts: Has anyone tried to use the hierarchy of ImageNet?

texts:
The classes of ImagNet have a hierarchy. Did anybody try to build a hierarchy of classifiers to use this fact?

Searching for "multistage classification" leads to different results.


texts:
I'm sure I have a small error here that I'm overlooking, but am having a tough time figuring out what I need to change.

Here is my code up until the error I'm getting.

# Load libraries
import pandas as pd
import numpy as np
from pandas.tools.plotting import scatter_matrix
import matplotlib.pyplot as plt
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analyisis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

# Load dataset
names = ['action','reject','approve','occ','loanamt', 'suffolk', 'appinc','typur','unit','married','dep','emp',yjob','self','atotinc','cototinc','hexp']

# from azureml import Workspace
# ws = Workspace(
#      workspace_id='',
#      authorization_token='==',
#      endpoint='https://studioapi.azureml.net'
# )
# ds = ws.datasets['loanapp_c.csv']

ds = pd.read_csv('desktop/python ML/loanapp_c.csv')
dataset = ds.to_dataframe()

I was running this on Azure and am now trying to do it locally. Here is the error I'm getting:

AttributeError                Traceback (most recent call last)
<ipython-input-3-b49a23658806> in <module>()
     32
     33 ds = pd.read_csv('desktop/python ML/loanapp_c.csv')
---> 34 dataset = ds.to_dataframe()
     35
     36 # shape

~/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py in_getattr_(self, name)
     4374 if self._info_axis.can_hold_identifiers_and_holds_name(name):
            return self[name]
  -> 4376 return object._getattribute_(self,name)
     4377
     4378 def _setattr_(self, name, value):

AttributeError: 'DataFrame' object has no attribute 'to_dataframe'

Not sure what I have wrong.


texts: Similarity measure based on multiple classes from a hierarchical taxonomy?

texts:
Could anyone recommend a good similarity measure for objects which have multiple classes, where each class is part of a hierarchy?

For example, let's say the classes look like:

1 Produce
  1.1 Eggs
    1.1.1 Duck eggs
    1.1.2 Chicken eggs
  1.2 Milk
    1.2.1 Cow milk
    1.2.2 Goat milk
2 Baked goods
  2.1 Cakes
    2.1.1 Cheesecake
    2.1.2 Chocolate

An object might be tagged with items from the above at any level, e.g.:

Omelette: eggs, milk (1.1, 1.2)
Duck egg omelette: duck eggs, milk (1.1.1, 1.2)
Goat milk chocolate cheesecake: goat milk, cheesecake, chocolate (1.2.2, 2.1.1, 2.1.2)
Beef: produce (1)

If the classes weren't part of a hierarchy, I'd probably I'd look at cosine similarity (or equivalent) between classes assigned to an object, but I'd like to use the fact that different classes with the same parents also have some similarity value (e.g. in the example above, beef has some small similarity to omelette, since they both have items from the class '1 produce').

If it helps, the hierarchy has ~200k classes, with a maximum depth of 5.


texts: I have just made an MNIST classifier from a tutorial. It has 1 layer, no hidden layers. It uses PyTorch's nn.CrossEntropyLoss as a loss function. Plotting my losses after training, it starts out at around 2.25 and goes down to 0.5. What does this number mean, in a colloquial sense? Does a loss of 0.5 mean that it's right 50% of the time?


texts:
My input data consist of list of list. Both list have dynamic length for every example like below.

X[0] = [[0, 1, 3, 5, 8, 26], [3, 4, 5, 7, 9, 15, 26, 30, 38, 39, 40]]
X[1] = [[1, 2, 4, 16, 19, 29, 48]]
..

My target data consist of dynamic length list like below:

y[0] = [5, 7, 8, 12, 15, 17, 29]
y[1] = [2, 4, 6, 8, 10, 11, 16, 18, 19, 29, 30, 33, 35]
..

I try to train LSTM model with given input X, predict Y.

If I concat each X list to one list and make padding operation LSTM accepts that data type. My example code is below:

X_train = sequence.pad_sequences(X_train, maxlen=padding_size)
X_test = sequence.pad_sequences(X_test, maxlen=padding_size)

model = Sequential()
model.add(Embedding(50, 10, input_length=X_train.shape[1], mask_zero=True))

if isBidirectional:
    model.add(Bidirectional(LSTM(lstm_layer_number)))
else:
    model.add(LSTM(lstm_layer_number))

if isDropout:
    model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) # binary crossentropy yields the best results

The problem is that I don't want to give input as a fixed size vector. Instead of given fixed size vector, I want to give input as a dynamic size and list by list.

How can I give these input as sequence(list by list) to LSTM ?

Any help is appreciated.


texts:
I was following this basic TensorFlow Image Classification problem, where images of flowers have to be classified into one of 5 possible classes. The labels in the training set are not one-hot encoded, and are individual numbers: 1,2,3,4 or 5 (corresponding to 5 classes). The final layer of the ConvNet however has num_class number of units.
Wouldn't there be a dimension mismatch while computing the loss, since you are finding the difference between a [num_class, 1] (predicted label) dimensioned vector and a [1, 1] (true label) dimensioned vector?
Does the Keras backend automatically convert the labels into one-hot vectors?
Thank you in advance!


texts:
I have some archive data that show the variation of a certain quantity X measured over 1-year intervals.

I'm now facing the problem of understanding how X correlates with another variable Y, that I can measure much more frequently, even daily.

So my question is, is it really necessary to collect data for Y so often? This seems such a waste of resources.

So what I would do is to collect data for Y only for 10% of the days of each year and I would select those days randomly. This way I'm sure that the observations are independent because the sample size is not too large and days have been selected randomly.

What do you think of that? Is that the approach you would follow as well?


texts: Should I pad zero or mean value in a convolution neural network? What about reflective padding?

texts:
In convolution layers, sometimes you need to pad some (usually 1 or 2 pixel) 0s at the edges of the original image before applying your convolution kernel (filter).

However, 0 has different meanings.

In the case of 0 - 255 gray scale images, 0 means black.

In the case of normalized images ( minus mean then divided by std), 0 is the mean value.

Should I always pad 0, or should I sometimes pad mean value ?

If in some cases I should pad mean value, what are those cases?

Edit:

And when should I use reflective padding?

Further more, is reflective padding always better than zero padding?


texts: Creating an Artifical Neural Network that produces a set of possible outputs

texts:
I'm new to machine learning, and was working on creating an ANN which would classify each observation to a certain value. I have worked primarily with the sigmoid function up to this point to get the probability of an observation (true/false or binary output).

In this instance, I want each observation to be classified to one of 5 values: 0, 1, 2, 3, 4. I am using the sklearn library's StandardScalar function to scale the input data. What would be recommended in terms of:

Best way to scale the output data, and
Appropriate activation function to use for the output layer.

Thanks!


texts: Looking for a good package for anomaly detection in time series

texts:
Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series?
There is a one class SVM package in scikit-learn but it is not for the time series data. I’m looking for more sophisticated packages that, for example, use Bayesian networks for anomaly detection.


texts: Features impacting individual predictions after training a classification model

texts:
I know my question might look odd but I just wanted to get some insights. Every prediction model will give us predictions for validation data set and it also can give/rank features based on their weight on predictions. I am looking for one step after that. Lets say we are dealing with employee resignation and we know the most important features are pay, latest promotion, job satisfaction and external job opportunities around that employees residence. Now, is there any way that we can say which of these features is the most important feature for an individual prediction? May be for one employee, Job satisfaction is the reason behind the leave but based on our feature importance, we need to focus on pay first. I just wanted to know if there is anyway that we can do some post processing on each individual predictions?

Thanks


texts: Difference usage of document level, sentence level and aspect level in sentiment analysis

texts:
I'm studying about Sentiment analysis. What is the purpose of using Document Level, Sentence Level and Aspect Level?

I read this definition in Web Data Mining book by Bing liu.

In this book it was stated that,

Sentence Level sentiment analysis is to classify a sentence to negative, positive, neutral class.

Document level sentiment analysis to classify a document.

Aspect Level sentiment analysis is to classify each aspect of entity mentioned in a review.

Thanks.


texts: LSTM to multivariate sequence classification

texts:
How can I train multivariate to multiclass sequence using LSTM in keras?

I have 50000 sequences, each in the length of 100 timepoints.
At every time point, I have 3 features (So the width is 3).

I have 4 classes and I want to bulid a classifier to determine class for sequence.
What is the best way to do so?

I saw many guides for univariate sequence classification but none for multivariate, and I don't know how to apply this on the multivariate case


texts: My small script on value alteration in columns of a data not working

texts:
I have a data set which has "Speed" as one of the columns (features). The column contains both zero and non-zero values. I want to randomly set 10% of the non-zero values to zeros. This will change the corresponding "class" label to zeros. I mean any value  set to zero, its corresponding class value will be zero as well. I have done this but it is give me errors. Though due to error, I cannot tell it will give me the update/result I want.

file_path = 'Processed_data/data1.csv'
df = pd.read_csv(file_path)
per_change = 0.1
attr = 'Speed'
target = 'Class'
df_spd = df[df['Speed'] > 0.]

num_rows_to_change = int(df.shape[0] * per_change)
num_with_zero_initial = df[df[attr] == 0].shape[0]
assert df_spd.shape[0] > num_rows_to_change, \
    'Number of rows with non-zero speed is less than 10% of the original   dataset.'
df_update = df_spd.sample(num_rows_to_change)
df_update[attr] = 0.
df_update[target] = 0.
df.update(df_update)
update_list = df_update.index.tolist()
num_with_zero_final = df[df['Speed'] == 0].shape[0]
assert num_with_zero_final == num_with_zero_initial + num_rows_to_change, \
    'Number of rows needed to change not equal to number of rows changed.'
df.to_csv('changed.csv')


texts: How to calculate customer purchase interval and predict next purchase in python?

texts:
Suppose we have a data set consists of columns

  TransactionId, CardNo, TransactionDate

then how can we calculate the customer purchase interval (means if customer A purchased on Jan 1st and after 10 days he again purchased, and then he again purchased after 15 days.) and how to predict the next visit of customer A by analysing the purchasing intervals of customer A.

Any help will be appreciated.


texts: How should continuous outputs be in convolutional neural network?

texts:
I have labeled faces images (label is the age -continuous value-) dataset and I want to construct a Convolutional Neural Network model to predict the age of a person.

I have the following questions.

How the label i.e. the age must be encoded ?
Which activation function to use ?


texts: Choosing between Regression and Classification

texts:
I want to build a model using a neural network that will be able to extract some features from landscape pictures.

In order to improve the efficiency of my model, I first want to extract the "vertical orientation" of the picture. This "vertical orientation" would be next a feature for my neural network.

Now, to compute this feature I see two solutions.

Build a Regression model that would return a result in degrees (0 to 180)
Build a Classification model that would return the class of orientation (ex: High, Medium High, Medium, Medium Low, Low)

Is there a way to decide which solution to use, or should I test both solutions to find the better one ?

Note :

The dataset is not a problem. I can have pictures labeled in degrees or in class easily


texts: Memory-efficient metric calculation for ultra high dimensional data

texts:
I am preparing for clustering the data which can be only represent as a extremely sparse binary vectors. Each of the objects is represented by a large set the binary features ($10^3$ ~ $10^6$), each identified by a 32-bit hash code. We can assume that in order to represent the data object without loss of information we need to see it as a vector of $2^{32}$ bits - in which only the bits at positions identified by hash are set to one.

The problem is, having two data objects, each represented by a sorted set of hashes, how can I calculate the distance between them without storing two $2^{32}$ bits long vectors in memory?


texts: Is it advisable or okay to view heatmap of categorical features?

texts:
As the title suggested. By default sns.heatmap will not include categorical features which are not factorized.

Since factorized categorical features do not behave like a scale (eg. Quality: 0-10), I assume we can't it's not advisable on categorical features?

Is there any other good practice that we can use to detect strong correlation between the categorical features (among themselves and with the label)?