texts: How would you optimize this Binary Text Classification model further? The data set is large (40000 texts)

texts:
I'm learning about tensorflow/keras since a couple months.
What are some methods to reduce overfitting in the later epochs or increase val. accuracy in general?
train data dim (40000,70) (reddit comments)
best results, after a lot of trial and error, so far with:
model = keras.Sequential()
model.add(keras.layers.Embedding(input_dim=14000,output_dim=400,input_length=70))
model.add(keras.layers.GlobalAveragePooling1D())
model.add(keras.layers.Dense(400, activation="relu"))
model.add(keras.layers.Dense(128, activation="relu"))
model.add(keras.layers.Dense(1,activation="sigmoid"))

optimizer = tf.keras.optimizers.Adam()
model.compile(optimizer=optimizer, loss="binary_crossentropy", metrics=["accuracy"])

fit_model = model.fit(x_train, y_train, epochs = 10, batch_size = 128, validation_data=(x_val,y_val), verbose=1)

It starts to overfit after 2 epochs(val acc=0.82).


texts: Image reconstruction using low-light components

texts:
Let's say we have a regular photo and three low-light photos illuminated in different colors.
Each pixel is a three-component vector $q=(R,G,B)$. Then $q_k^{A}$ is the $k$-th pixel of the regular photo and $q_k^{B}$ $q_k^{C}$ $q_k^{D}$ be the $k$-th pixel of the three low-light versions.

The task is to reconstruct the regular photo from the three low-light photos where:

$q_k^{A} = F^{A}q_k^{B} + F^{C}q_k^{C} + F^{D}q_k^{D} + q_{const}$. Clearly, $F^A, F^{B}, F^{C}$ are $3 \times 3$ matrices and $q_{const}$ is a vector.

The task is to perform least-squares fit for the 30 components (three matrices and the vector). Specifically, we should minimize:

$S = \frac{1}{MN}\sum\limits_{k=0}^{MN-1}{||-q_k^{A} + F^{A}q_k^{B} + F^{C}q_k^{C} + F^{D}q_k^{D} + q_{const}||^2}$

What is the proper way to approach this problem? I have implemented linear regression of different types before, but in this case I am not sure how to proceed.


texts:
I would like to determine features importance in several models:

support vector machine
logistic regression
Naive Bayes
random forest

I read that I will need an agnostic model, so I have thought to use performance_importance (in python).
My features look like

Text (e.g., The pen is on the table, the sky is blue,...)
Year (e.g., 2019, 2020,...)
#_of_characters (e.g., 34, 67,...): this value comes from Text
Party (e.g., National, Local, Green, ...)
Over18 (e.g., 1, 0, ...) : this is a Boolean variable

My target variable is Voted.
In the pre-processing phase, I am using BoW and TF-IDF for Text, OneHotEncoder for Party, SimpleImputer for numerical.
Using the following:
from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

result = permutation_importance(clf, X_test, y_test, n_repeats=5, random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
            vert=False, labels=X.columns[sorted_idx]);

I am getting a similar output like the below(I forgot to include Over18, but it is just to give an idea of the output):

Although I have difficulties in interpreting the results, especially circles and negative values, I would like to understand if, in case of text classification, it makes sense to have Text for the importance and not, for example, the single words (unigrams, bigrams,...).
For instance, in my example, I have ['The','pen','is','on','table','sky','blue']. Would it make more sense to understand how much each word contributes to the model instead of Text, or this is just considered in Text (where there are many words that contributes to the model), which is the most significant feature in the model?
UPDATE: for the different features I am using the following pre-processors:
categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')

numeric_preprocessing = Pipeline([
    ('imputer', SimpleImputer(strategy='mean')
])

# CountVectorizer
text_preprocessing_cv =  Pipeline(steps=[
    ('CV',CountVectorizer())
])

# TF-IDF
text_preprocessing_tfidf = Pipeline(steps=[
    ('TF-IDF',TfidfVectorizer())
])

and then
preprocessing_cv = ColumnTransformer(
    transformers=[
        ('text',text_preprocessing_cv, 'Text'),
        ('category', categorical_preprocessing, categorical_features),
        ('numeric', numeric_preprocessing, numerical_features)
], remainder='passthrough')

clf_nb = Pipeline(steps=[('preprocessor', preprocessing_cv),
                      ('classifier', MultinomialNB())])


texts: How can I preprocess text to feed into a SVM?

texts:
I am using an IMDB dataset which contains reviews of the movies in the column text and the rating 0 or 1 in the column label. I am preprocessing the text using Tfidf using sklearn.
The code for the above statement
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer=TfidfVectorizer()
X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_test['text'])

When I am trying to use the data for an SVM, using a linear kernel and then fitting it into the model using
classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, df_test['label'])

I am getting the error
ValueError: Found input variables with inconsistent numbers of samples: [40000, 5000]
df_train is of the shape (40000,2)
df_test is of the shape (5000,2)
I am able to overcome this problem by using only 5000 values of df_train using
df_train.loc[:4999,'text']

but this defeats the purpose of having a bigger training dataset.
My question is how can I use the training dataset that will retain it's number of values?


texts: How to extract important text markers from samples to identify patterns?

texts:
Problem
I have collected a decently large set of movie trailer titles from various Youtube trailer channels. I'd like to extract or infer the movie title and release year from this set in some way to look up additional information about it via a browser extension.
Attempted Solutions
Regex. I've developed a fairly successful regular expression to work with the most common trailer channels, but new formats tend to pop up every now and then which renders the results invalid or incomplete. I've added a sample of my list (200k+) with raw titles from Youtube and manual extraction by myself in the parsed title and parsed year columns. There tends to always be a pattern to these titles such as "title is at the start and all caps", "year is usually in parentheses", and many others.
Question
I'm looking to extract EXACT text fragments, not summarizations. In the below examples, casing is maintained and the extraction occurs at a specific index in the source string. I'm trying to "understand" where these indices are based on training/sample data.
How would I go about extracting this index information automatically?
Sample Data

raw_title
parsed_title
parsed_year

MARVEL'S SHANG-CHI Official Trailer (2021)
MARVEL'S SHANG-CHI
2021

ARTHUR AND MERLIN: KNIGHTS OF CAMELOT Official Trailer (2020) Action, Adventure
ARTHUR AND MERLIN: KNIGHTS OF CAMELOT
2020

HIDDEN IN PLAIN SIGHT Official Trailer (2019) Thriller Movie
HIDDEN IN PLAIN SIGHT
2019

MORTAL KOMBAT "Scorpion VS Sub-Zero" Opening Scene (2021)
MORTAL KOMBAT
2021

SHANG-CHI AND THE LEGEND OF THE TEN RINGS Trailer (2021) Marvel Movie HD
SHANG-CHI AND THE LEGEND OF THE TEN RINGS
2021

WHITNEY Trailer #2 (2018) Whitney Houston Documentary, Music Movie
WHITNEY
2018

THEY Trailer (Cannes 2018) Rhys Fehrenbacher, Koohyar Hosseini Thriller Movie
THEY
2018

RAMPART Movie Trailer # 1  (COP Drama - Woody Harrelson - 2012
RAMPART
2012

Mainstream Teaser Trailer (2021) - Movieclips Trailers
Mainstream
2021

Mortal Kombat - First Seven Minutes (2021) - Movieclips Trailers
Mortal Kombat
2021


texts: Information bottleneck and deep neural network

texts:
I learned about "the information bottleneck view of deep learning."
But in a nutshell, what does this tell us?
I don't see what the role is of depth in this approach as long as it is larger than 2 or 3.
Is there a rigorous theory? Or just some hypothesis or heuristic explanations on deep neural net?
I saw the author's talk on YouTube.
But, probably my ignorance, I don't really get the main point and the implication is.
I can see a lot of explanations on graphs on the video, but honestly, I don't get it.
Any comments, suggestions, opinions will be very appreciated.


texts: Patch wise training vs Full Convolutional Training in semantic segmentation

texts:
As mentioned in the title, what are those 2 methods? I already checked this question: Patchwise and Full training, (and the mentioned paper) but i can't really understand the meaning and the process of those 2 methods.

What's the difference between the two? When should i prefer one instead of the other one?


texts: Are jobs the only way out for data scientists?

texts:
I have reached this point in my data sceince journey that I am left wondering if jobs are the only way out for me...

I am highly skilled in Hadoop, Machine Learning and now getting into Deep Learning algorithms...

I am bored with having to be employed, I want a way out...any ideas? I would still like to use my skills in my life but what other options exist out here...

I am open to ideas and collaborations....


texts: Can features that are the same in every sample contribute to learning?

texts:
For simplicity, let's say that I am monitoring 4 sensors for an ongoing metric.
The first column is the sensor ID and the second column is the sensor type.
[
  [
    [0, 0, 0.123],
    [1, 0, 0.456],
    [2, 1, 0.789],
    [3, 1, 0.555]
  ],
  [
    [0, 0, 0.987],
    [1, 0, 0.654],
    [2, 1, 0.321],
    [3, 1, 0.666]
  ],
  [
    [0, 0, 0.591],
    [1, 0, 0.824],
    [2, 1, 0.760],
    [3, 1, 0.888]
  ]
]

If the first two columns are always the same values, will a CNN or an LSTM be able to learn from these columns or are they just redundant?
In my mind, the sensor ID could correspond with a postion on the map where different metrics are observed. Or the sensor type could correspond with some sort of sensitivity in the metric. But am I just kidding myself if they are the same in every sample?
I don't want to provide unnecessary dimensionality to the model.


texts: One-hot encoding with values other than 1

texts:
I was thinking if I have an input which has 36 possible values, and I make it as 36 inputs where exactly one of them is non 0, what is optimal value for each of the non 0 inputs?

It may be:

[1, 0, 0,....,0]
[0, 1, 0,....,0]
[0, 0, 1,....,0]

Or:

[36, 0, 0,....,0]
[0, 36, 0,....,0]
[0, 0, 36,....,0]

Or even:

[6, 0, 0,....,0]
[0, 6, 0,....,0]
[0, 0, 6,....,0]

In order this feature to have same impact on the network as any other feature with N(0,1) distribution by keeping in mind I will use L1 or L2 regularization

So for each weight on a normal input I will have 36 weights on those one-hot inputs, so regarding L1 those weights should be 36 times smaller in order to have same impact? Don't they?

But then their total impact in affecting result is small since only one of them is being multiplied by 1 and included in calculation...

So if you would be kind to explain this and convince me to use 1s instead 6s or 36s, please


texts: How to determine a sufficient number of training examples for a linear regression classifier?

texts:
How do we determine a sufficient number of training examples for a linear regression classifier?
What kind of behaviors might we expect if we use too few training examples?


texts: Are interactive command lines from orange3 possible

texts: I wonder whether orange3 allows to get commands that are running "behind" a certain statistical procedure? This would be like in R, when using RCmdr.


texts: Verifying business ideas based on keywords derived from a description of the business idea

texts:
I want to find keywords for a business idea. The problem is that I have a hard time summarizing the business idea to one to three terms that would describe it.

Is there a (machine learning) tool that takes a description as an input and outputs - intelligently - categories and/or explicit search terms that fit the business idea most likely?


texts: Percentile as a threshold for Anomaly Detection?

texts: I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is an outlier or not. What is the connection between percentile value and threshold and how can we decide if a point is an outlier or not using a threshold value?


texts: How to program derivatives for recurrent weights

texts:
I understand that the formula for the gradient of W and U is this:

How might I go around programming ds/dst-1 for example?


texts: early-stopping changes final epoch in training each time

texts:
I am training a CNN built using transfer learning with a VGG16 network as pre-trained model, and in the training I am using early-stopping as regularization technique. I have run several time the training, and what I see is that early stopping stops the training process everytime at a different epoch, so for example if I train the CNN the first time I arrive until the epoch 17, then another day I train again the network and during training it arrives at the epoch 8, and the epoch at which arrives it is different everytime.

Is this normal? I don't understand why everytime it changes the epoch at which it arrives.

Can someone explain this to me? Thanks in advance.

[EDIT] My code for the training is the following:

from keras import callbacks
# fit the transferNet on the training data
stopping = callbacks.EarlyStopping(monitor='val_acc', patience=3)

steps_per_epoch = train_generator.n//train_generator.batch_size
val_steps = test_generator.n//test_generator.batch_size+1

try:
    history_transfer = transfer_model.fit_generator(train_generator,
epochs=100, verbose=1, callbacks=[stopping],\
                    steps_per_epoch=steps_per_epoch,\
                    validation_data=test_generator,\
                    validation_steps=val_steps)
except KeyboardInterrupt:
    pass


texts: What is the correct terminology for neural network architectures that expand in their internal layers? (The opposite of a "bottleneck")

texts:
I am aware that an auto-encoder contracts in its hidden layers to form a bottleneck.
In contrast to this, is there a good name for the kind of cell, block or architecture that expands in its internal layer(s) and then contracts in its output/interface layer(s).
For example, Deep Equilibrium Models have a residual cell that expands in its internal layer and then contracts at its interface.


texts: Should I perform cross validation only on the training set?

texts:
I am working with a dataset that I downloaded from Kaggle. The data set is already divided into two CSVs for Train and Test.

I built a model using the training set because I imported the train CSV into a Jupyter Notebook. I predicted using the Train CSV itself. I would like to perform cross validation. Should I perform cross validation on the train CSV and split it into two parts again Train and Test? Or, should I import a new CSV file Test and combine both CSVs into ONE?


texts:
My company tracks objects.  I have developed an algorithm that tries to identify objects that are attached to each other.  It does this in real time and starts trying to identify objects after roughly 5 minutes of travel.

Currently this uses some basic time filtering plus a Hausdorff distance algorithm to ensure it tracks objects that leave at roughly the right time and follow roughly the right path/trajectory.  Note that these tracked object will certainly report their time & position at irregular intervals. (Between 30s and 5minutes per update)

This alone achieves a great level of accuracy.  However in certain cases four objects will leave at roughly the same time.  Two primary objects each with an attached secondary object (i.e. 2 + 2). In this case the system fails.

I thought that if I could somehow include time in the Hausdorff distance it may help improve the accuracy.

Is this actually a viable solution and how might I modify my Hausdorff algorithm to also include a time series parameter?

Failing this is there a suitable machine learning algorithm that would be worth looking at?


texts: How to know the probability of correctness of a test data in a Binary Classifier

texts:
I have written a sequential classifier script using Keras, Tensorflow. Its a binary image classifier that predicts the class, given the directory path of a sample image.

I want to implement a probability feature in my script, where including the class, the correctness probability of my classifier's prediction is given too. for eg:

INPUT

img = image.load_img('dataset/test/sample/fff.jpeg', target_size = (img_width, img_height))
img = image.img_to_array(img)
img = np.expand_dims(img, axis = 0)

prediction = model.predict(img)

OUTPUT (something like)

Prediction: 80% seems like it being Class A

In other words, I want to display my classifiers accuracy for that particular image.

P.S I am sorry if the question seems a little abrupt. I have tried asking it elsewhere too but either It gets closed due to lack of information, Or its misunderstood as "Classifier training accuracy" by google as its hard for me to frame it.


texts: How to model a binary classification problem in a evolving environment

texts:
Suppose you have a problem with two classes YES or NO. While the YES class is fixed, in the sense that the observations do not evolve, the observations of class NO may evolve during time and may change to the class YES. I would like to predict the probability of a given observation change to YES.

What kind of ML models could be used here? Could Bayesian Networks be a solution?

I would also like to observe what changed in the variables (or which variables were most important) that turned a NO into a YES.

Thank you


texts: SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?

texts:
I am using LogisticRegressionWithLBFGS to train a multi-class classifier.

Is there a way to get the probability of all classes (not only the top candidate class) when I test the model on new unseen samples?

P.S. I am not necessarily obliged to use the LBFGS classifier but I would like to use the logistic regression in my problem. So if there is a solution by using another LR classifier type, I would go for it.


texts: Question about train example code for TensorFlow

texts:
I am trying to learn TensorFlow, and I could understand how it uses the batch in this example:

cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
sess.run(tf.initialize_all_variables())
for i in range(20000):
  batch = mnist.train.next_batch(50)
  if i%100 == 0:
    train_accuracy = accuracy.eval(feed_dict={
        x:batch[0], y_: batch[1], keep_prob: 1.0})
    print("step %d, training accuracy %g"%(i, train_accuracy))
  train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

print("test accuracy %g"%accuracy.eval(feed_dict={
    x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))

My question is, why it get a batch of 50 training data, but only use the first one for training. Maybe I did not understand the code correctly.


texts: Analyze performance Poisson regression model on a time series(count forecasting)

texts:
I have tried to build a model to forecast the count of a particular variable.The model that was used for the purpose was poisson .Unfortunately ,i don't have enough stat knowledge to analyze the model performance .If somebody can provide some insights as of how the model is performing,as well as some tweaks to improve the model performance will be greatly helpful. I am also willing to try out other models if it performs better.

I am using python with the statmodels package to build the model.

  Attaching a graph which shows the fitted and the actual
  values(Green shows the actual values and Blue shows the fitted
  values)

Also,providing the summary() output of the model

                 Generalized Linear Model Regression Results
==============================================================================
Dep. Variable:         Work_Item_Type   No. Observations:                  581
Model:                            GLM   Df Residuals:                      574
Model Family:                 Poisson   Df Model:                            6
Link Function:                    log   Scale:                             1.0
Method:                          IRLS   Log-Likelihood:                -16752.
Date:                Mon, 22 Feb 2016   Deviance:                       31268.
Time:                        21:59:12   Pearson chi2:                 1.05e+05
No. Iterations:                     9
===============================================================================
                  coef    std err          z      P>|z|      [95.0% Conf. Int.]
-------------------------------------------------------------------------------
Intercept       2.8492      0.051     55.426      0.000         2.748     2.950
Weekday        -0.2066      0.032     -6.446      0.000        -0.269    -0.144
day_of_week    -0.0926      0.007    -13.367      0.000        -0.106    -0.079
wom             0.1122      0.007     16.996      0.000         0.099     0.125
week           -0.0411      0.001    -53.597      0.000        -0.043    -0.040
TimeDelta       0.0001    5.1e-05      2.933      0.003      4.96e-05     0.000
month_of_yr     0.2192      0.004     60.981      0.000         0.212     0.226
===============================================================================

Also attaching a sample of the dataset used

clear_date  Count_Work_Item_Type
7/7/2014    1
7/10/2014   1
7/11/2014   5
7/17/2014   2
7/22/2014   1
7/24/2014   1
7/29/2014   3
7/30/2014   4
8/13/2014   1

Since i had only the date and the variable to be forecast i created a bunch of other variables like

Weekday(binomial)?
Day of week
Week of Month
Week
Time Delta (Starts from 0 increment by one until end)
Month of Year

Also,i haven't done any kind of transformation on the variables.
Please do comment if you need additional information:
Thhanks


texts: how to see decision tree when running in anaconda?

texts:
In Jupyter displaying the DT is done as follows:

# Display in jupyter notebook
from IPython.display import Image
Image(filename = 'tree.png')

How to see DT in anaconda?


texts: Training xgboost model with more data having different characteristic

texts:
I have trained my model for ECG data which has 8528 ECG files having length 30s and sample rate 300 so total file length in csv is 9000. Data has four classes i.e. A, N,O and ~

This model is working fine.

Now I want to retrain the model using few more ecg files. Which has length 42s and sample rate 360. So total csv file of this ecg signal has length 15127.

I trained model with A file having length 15127, then I tried to predict same file with trained model, but it predicts class value N.

Can anyone visualie how Xgboost will learn with added such file having signal length more? Does such file encountered as outlier?

If I train model with more A having lenth 15127 length will model be able to predict it correctly?

Let me know if any information required.

Here is sample training code :;

dtrain = xgb.DMatrix(train_data,label=train_labels)
# cross_validate(dtrain)
print('training model.....')
param = {'num_class':4,'objective':'multi:softmax','eval_metric':['merror'],'max_depth':7,'eta':0.04,'subsample':0.8,'min_child_weight':0.5,'max_delta_step':7,'gamma':2,'lambda':10,'colsample_bytree':0.5}
model = xgb.train(param,dtrain,420,[(dtrain,'train')],verbose_eval=20)
print('saving model....')
model.save_model('xgb_model.bin')
print('done!')


texts:
What is the basic philosophy behind feature selection and modelling? How do you actually start? Could you please share your real (practical) inputs?

Bit of background:
I am actually trying to analyse incoming service desk requests (emails) and predict/assign it to the concerned department (team).

I have identified 2 features, but they are giving ~70% prediction accuracy, because of few ambiguous entries. So, need to add more features to strengthen my model.

I am sure your inputs will help newbies (like me) to get the initial idea.


texts: How to find unknown number of clusters in circular data?

texts:
I have some 1 dimensional data. Each record in the data is a specific time of the day. In order to cluster it I projected the data onto a circle of radius 1 unit. Now I need to find clusters in this data. The number of clusters are unknown and it is preferred to find clusters with high density of records in them. By density I mean that a large volume of records should be packed in a small space.

How should I go about finding clusters in the above mentioned data?


texts: Converting a nominal attributes to numerical ones in data set

texts:
I'm using the NSL-KDD data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the get_dummies method in python and the NominalToBinary method in WEKA, but the problem is that some nominal features contain 64 values so the conversion increases the dimensionality of the data a lot, and this can create problems for the classifier.

My question is if I can convert the nominal attributes by establishing a correspondence between each category of a nominal feature and a sequence of integer values, for example protocol_type {tcp=0, udp=1, icmp=2...etc}? Would this alter the credibility of the resulted data set?


texts: Distribution of Regression Residuals: Is this a normal distribution?

texts:
I've created a histogram as well as a QQPlot from the residuals of my Regression Model:
Mean: 0.35
Standard Deviation: 18.14

Judging from these plots, is it okay to say that my residuals are normally distributed?
Or what else can I draw from these plots?
Update: Created the Histogram using
ns.distplot(x, hist=True)

Here's the result:


texts: Increasing minNumObj increasing accuracy in decision tree

texts:
I have been using a J48 classifier in weka and have noticed that increasing
minNumObj -- The minimum number of instances per leaf leads to a small accuracy increase.

-M      Result.     Size    Num Leaves
2       73.8281 %   39      20
3       74.2188 %   39      20
4       74.4792 %   37      19
5       74.6094 %   25      13
6       74.2188 %   23      12
7       74.2188 %   23      12
8       74.349  %   23      12
9       75.2604 %   29      15
10      75.5208 %   29      15
11      75%         23      12
12      76.3021 %   23      12

However in several examples such as :

http://ww.samdrazin.com/classes/een548/project2report.pdf

The opposite its shown , which minNumObj increasing lowering the accuracy. The confidence factor was held constant at 1.0 to minimize post-pruning. Cross validation folds for the testing set (crossValidationFolds) was held at 10.

My results were made with a confidence factor of 2.5 but the difference between 2.5 and 1 is minimal.

https://pdfs.semanticscholar.org/9984/a7d06e04a347718cb8c7f645b72195bb11ce.pdf

See section 5.3. Measuring Performance: Precision and Recall

Why is the accuracy in my data going up and not down with an increase of minnumobj?


texts:
I'm newbie into data science, and I had some problems dealing with my project. I'm trying to visualize multidimensional data into 2D after clustering with using a lot of methods. (kmeans, DBSCAN, OPTICS, agglomerative, spectral...)

I have multidimensional data. (11 columns - attributes , 150K rows - number of data). It is slightly sparse-alike data, for example, which means one datum has numeric values like (0, 0, 6.5, 0, 0, 7.5, 0, 0, 4.5, 0, 0)

So, each datum has approximately 2~5 non-zero attribute values...

Below is not exactly same with my project, but it's similar.

https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html

But, as I'm new to this, I am curious about the sequence of PCA and Clustering. I think there are 2 scenarios.

[1. Do clustering before PCA]

That means, I am using PCA just for visualization. But I have a question. In that case, If I process clustering with raw data, are all clustering algorithm (mentioned above) fit to my data type well.

[2. Do clustering after PCA]

In this case, I have other problems. My data's importance of components are like below.

                          PC1    PC2    PC3    PC4     PC5     PC6     PC7    PC8     PC9    PC10
Standard deviation     1.4173 1.1836 1.1141 1.0108 0.99109 0.95231 0.89091 0.8456 0.71542 0.64610
Proportion of Variance 0.2009 0.1401 0.1241 0.1022 0.09823 0.09069 0.07937 0.0715 0.05118 0.04174
Cumulative Proportion  0.2009 0.3410 0.4651 0.5673 0.66551 0.75620 0.83558 0.9071 0.95826 1.00000

As far as I've understood about visualizing multivariate data into 2D, I have to choose 2 PCs.(e.g.> PC1, PC2). However, my data's PV is slightly lower than I'd expected.

So, is it okay if I choose (PC1, PC2) to coordinates to be clustered and process clustering? Also, can I choose other PCs (e.g. PC5, PC8) to coordinates to be clustered?


texts: How to improve an existing (trained) classifier?

texts:
I have a Classifier which I have trained and tested on a small dataset - receiving solid results, though I wish to improve them. If I understand correctly one way of doing so is to add more data to obtain a more precise classification rule.

When doing this should I add data to both the training set and the test set? or should I add only to the training set? or maybe I should I create new training and test sets from the 'new dataset'? (new = the old data + the new data).


texts: Extracting and tagging/labeling text from pdfs

texts:
I'm trying to extract paragraphs from 230 pdfs that have similar formats, and then label the paragraphs based on the pdf's section headings. Normally, I'd convert the pdf to txt files or extract the text using pdftools on R, but this would prevent me from distinguishing the text in headings -- which is bold, has a certain color -- from the text in the body that I want to "tag". I have access to Adobe Acrobat Pro, so I can convert the pdfs to .txt, .xml, .html, .rtf, or .docx files.

Does anyone have any suggestions on the best way to proceed?
If there isn't a straightforward answer to the first question, is there one file format among the ones listed above (aside from .txt, which would strip the formatting) that would be easier to parse than others?

Thanks!


texts: How should I expose data from my app for data scientists?

texts:
I'm the product manager for an online app.  I'm currently researching a new feature where our users will be able to access "all their raw application data".  This data is likely to be used by data scientists, it's also likely to be loaded into BI tools.  The datasets will contain up to a few million rows.

How should I actually expose the data in a practical sense?

For example:

Online datasource like Amazon Redshift
Some other RDBMS available online (e.g. a dedicated postgres installation)
CSV files available on S3
CSV files available for download in a web interface
Dumps into Google sheets
Doesn't matter as decent data scientists can easily handle and automate anything


texts:
I have a database of 3190 instances of DNA consisting of 60 sequential DNA nucleotide positions classified according to 3 types: EI, IE, Other.

I want to formulate a supervised classifier.

My present approach is to formulate a 2nd order Markov Transition Matrix for each instance and apply the resulting data to a Neural Network.

How best to approach this classification problem, given that the Sequence of the data should be relevant? Is there a better approach than the one I came up with?


texts: Reason for Training and test loss sudden increment after some epochs keras

texts:
We know that if training and test loss are different from each other, our model is over-fitting. However, if both get high after some epochs, how can we justify it?

One way to solve it is to reduce the learning rate to 0.0001.

But, actually I would like to know the theoretical reason behind this sudden increment.


texts: Should unique vectors (SIFT descriptors) be used in K-Means Clustering?

texts:
I'm doing image classification by extracting SIFT features, clustering them and then finding BOVW histogram and classifying.

I have around 180 training images from which I'm extracting SIFT descriptors. I need to cluster these features using k-Means clustering. Now, among the SIFT descriptors, some are duplicate. Before applying K-Means, should I remove these duplicate vectors or should I not? Or does it not make any difference?

Thanks!


texts: Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

texts:
Reference
I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning?
Also, what is learning decay and learning offset, in this context and in general?


texts:
Let's say I have a dataset with pricing information for the same flight during the past year. So, for a flight departing on day D, I have the available price from D-130 up to D (departure day). Then the same for flights on the other 365 days of the year for the same departing time every day. Does this make sense? I want to see if today's price for any departure day in the future is higher or lower than expected, hence if it's expected to rise or not.

Plotting this data on a chart, the X axis are the days prior to departure, and the Y axis the price, I get the following:

How can I shape the data so I can train a model for price prediction? I can't see a clear trend. The price just below 90 means that it's the max price and it's not discounted.


texts: Pre-processing data to make predictions on deployed Sklearn model

texts:
I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...)
I have used pickle to Serialize my model and save it as .sav file. I would like to use it in REST API created with Flask. I am able to load it and all, but the problem is that I have used some feature engineering, feature scaling, and feature encoding using pd.get_dummies during the pre-processing phase before training my model. I assume the data that get's passed in the POST request of my REST API should also be pre-processed to be similar to the data that has been used to train the model (otherwise I would get the error : ValueError: Number of features of the model must match the input. ), so I created a function to do that. The problem lies in the encoding of the features. The new features created by encoding on the training dataset (pandas dataframe with multiple rows of data) and the input data (only one row) are different. How should I approach this in your opinion ?
Here are a two images that illustrate my problem,
This is an input example that has been pre-processed to get the predicted output by the model :

And here's an example from the training dataset that has been pre-processed :

Here's the code that I plan on using to transform the data that gets sent to my API :

def input_data_transform(inp):
    sc_X = StandardScaler()
    numericals =  pd.DataFrame(sc_X.fit_transform(inp[['carat','depth','x','y','z','table']]),columns=['carat','depth','x','y','z','table'],index=inp.index)
    inp[['carat','depth','x','y','z','table']] = numericals[['carat','depth','x','y','z','table']]
    inp =  pd.get_dummies(inp)
    return inp

In your opinion how should this be approached ? Is there a way to resolve this problem for most cases ? A simple and optimized solution ?


texts: Multidimensional scaling producing different results for different seeds

texts:
I took the data from here and wanted to play around with multidimensional scaling with this data. The data looks like this:

In particular, I want to plot the cities in a 2D space, and see how much it matches their real locations in a geographic map from just the information about how far they are from each other, without any explicit latitude and longitude information. This is my code:

import pandas as pd
import numpy as np
from sklearn import manifold
import matplotlib.pyplot as plt

data = pd.read_csv("european_city_distances.csv", index_col='Cities')

mds = manifold.MDS(n_components=2, dissimilarity="precomputed", random_state=6)
results = mds.fit(data.values)

cities = data.columns
coords = results.embedding_

fig = plt.figure(figsize=(12,10))

plt.subplots_adjust(bottom = 0.1)
plt.scatter(coords[:, 0], coords[:, 1])

for label, x, y in zip(cities, coords[:, 0], coords[:, 1]):
    plt.annotate(
        label,
        xy = (x, y),
        xytext = (-20, 20),
        textcoords = 'offset points'
    )
plt.show()

Most of the cities seem to be around the correct general location relative to each other, except a few infractions - Dublin is too far away from London, Istanbul is in the wrong location, etc. However, if I give a different random_state value, it produces a different "map". For example, random_state=1 produces the following map, where many of the cities do not seem to be around the correct general location relative to other cities:

What I don't understand is, dimensionality reduction methods are not supposed to have randomness associated with them, and thus should not give different results for different seeds. But it does here; so what does it mean?

The documentation of the sklearn.manifold.MDS function states that random_state is "the generator used to initialize the centers". So, in particular, I guess what I'm asking is, whatever initialization of the centres we choose, shouldn't all of them lead to one unique result?

I get a much more "accurate" map (to my eyes at least) by giving the following hyperparameter values:

mds = manifold.MDS(n_components=2, dissimilarity="euclidean", n_init=100, max_iter=1000, random_state=1)


texts: Filling missing values for important features

texts:
So I have this dataset from census the census bureau database which contains 40 attributes and a target column specifying if the total income is >50k.
P.S : this is not the known ADULT dataset (which is a lot cleaner), the difference is that mine wants to predict the total person income rather than the adjusted gross income.

My problem is that some of the attributes such as : class of worker (meaning if he's self employed, local government...),  major occupation and and industry codes have 100 000 missing values knowing that the total length of my dataset is 150 000.

Now it bugs me to simply drop those column because it seems to me like an important information to know if we want to predict the total income.

What should I do in such a position ?

Thanks !


texts:
I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day:

day
store_a
store_b
store_c

2021-01-01
100
200
300

2021-01-02
110
220
290

2021-01-03
50
110
170

2021-01-04
NAN
220
290

2021-01-05
7
16
NAN

2021-01-06
90
NAN
NAN

I'm using the IterativeImputer class from scikit-learn, this method of imputation puts aside one column at each step and train an estimator on the other columns to predict the column that was put aside.
My question is: should I transpose my dataframe or not?
If I keep my dataframe (1 line = 1 day, 1 column = 1 store), this means that we can completely predict the number of visitors in a store in a particular day just by looking at the other stores.
But if I transpose my dataframe (1 line = 1 store, 1 line = 1 day), this means that we can predict the number of visitors just by looking at the history of one store.
I guess one simple way to check if I should transpose is comparing the RMSE of the two methods but I wanted some explanation instead of "method A works better, move along."


texts: Multiclass Classifier comparison decision regions

texts:
How can I get the very same effect of this tutorial in Scikit Documentation with more than 2 classes?
Let's say we'll keep only the first dataset (the linear separable one) and substitute it with n_classes=3 like in the following snippet:
X, y = make_classification(n_features=2, n_redundant=0, n_informative=2,
                          random_state=1, n_clusters_per_class=1, n_classes=3)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [linearly_separable]

I'd like to get the very same effect with plotted the probabilities in a colormap.
Thank you!


texts: Difference in labelling and normalizing train/test data

texts:
I am working on a dataset comprised of almost 17000 data points. Since it's a financial dataset and the components are many different companies, I need necessarily to split it by date. Therefore, supposing I have 10 years of data, I am training over the first 8 years and testing over the remaining 2. This approach I am pretty sure is consistent with the classification problem I need to do.

I am using LSTM network for predicting the direction of financial returns, depending on a bunch of features which are derived from companies' financial statements. Starting from the fact I am obtaining training accuracy greater than test accuracy with almost any architectures and hyperparameter configuration, I suppose there is something wrong in the way I have manipulated the dataset.

Here comes my concerns. I labelled my dataset looking at the median returns and putting 1 if the return for a single data point (company value at a specific date) is above such median, 0 otherwise. Am I correct if I compute two different medians? So that I labelled the training set using its median return and in the same way the test set using its own median return? Should I compute the median over the entire dataset, label it and then splitting?

Moreover, I scaled the training data to be in a range of (0,1). Should I do the same kind of normalization with my test set? I did it, but I wasn't sure about it.

It's kind of my first application of neural networks and I need those clarification about hwo treating the dataset, without influencing the results.


texts: Why does a machine learning model of English can not be directly applied on right to left languages, like Arabic Hindi, and Urdu

texts:
I have a Logistic Regression model that works fine when used with an English Dataset. But, when I pass a same format data set in a different language, (Language is URDU which is right to left and UTF-8 encoded ) the model does not predict.

I understand that the preprocessing differs for URDU, Arabic, and Hindi but even if it has been performed separately, still the model is unable to predict at all.

I would like to know why the same model can not perform on different languages directly. Good or bad results are another topic. How can I make a model which is scalable enough to perform on both English and Arabic? What are the constraints in doing so?


texts: Is there a node embedding method that is portable between graphs?

texts:
I have several graphs whose nodes I want to encode to later process them with a neural network. I want the embeddings to take into account the topology of the graph around the nodes I am embedding, such that nodes from different graphs but with a similar topology have close embeddings.
For instance, on the figure below, node A from graph 1 and node B from graph 2 should have close embeddings because the topology of the graph around them is similar.

Moreover, I want to be able when I get a new graph to quicly compute the embeddings of its nodes, and make sure that they are coherent with the embeddings I got earlier. For instance, if I get a third graph, with a node C that has a topology around it similar to node A and B, I want to be able to get its embedding (that should be close to A and Bs embeddings) without having to reprocess all the graphs.
To sum up, I am looking for a (deterministic ?) approach that would allow me to encode nodes, and would yield similar results for nodes from other graphs but with a similar topology.
Is there any embedding method that allows that? From my understanding, most node embedding methods (like GraRep, HOPE, node2vec..) are not portable to other graphs because they are not deterministic, and could yield different results after training with nodes from different graphs (even if these nodes have similar topologies).


texts: Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same weights with different hyperparameters), and Test set is used for evaluating our model". If true, can someone explain the whole process in a little more detail. TIA.


texts: Linear Discriminant Analysis, which parameters can be tunned in cross validation set up?

texts: I am implementing Linear Discriminant Analysis in R, which parameters can be tunned in cross validation set up? In regularized mode called penalizedLDA there are parameters which are optimised but I want to know which parameters are turned in case of simple LDA method?


texts: Agent always takes a same action in DQN - Reinforcement Learning

texts:
I have trained an RL agent using DQN algorithm. After 20000 episodes my rewards are converged. Now when I test this agent, the agent is always taking the same action , irrespective of state. I find this very weird. Can someone help me with this. Is there a reason, anyone can think of why is the agent behaving this way?

Reward plot

When I test the agent

state = env.reset()
print('State: ', state)

state_encod = np.reshape(state, [1, state_size])
q_values = model.predict(state_encod)
action_key = np.argmax(q_values)
print(action_key)
print(index_to_action_mapping[action_key])
print(q_values[0][0])
print(q_values[0][action_key])

q_values_plotting = []
for i in range(0,action_size):
    q_values_plotting.append(q_values[0][i])

plt.plot(np.arange(0,action_size),q_values_plotting)

Every time it gives the same q_values plot, even though state initialized is different every time.Below is the q_Value plot.

Testing:

code

test_rewards = []
for episode in range(1000):
    terminal_state = False
    state = env.reset()
    episode_reward = 0
    while terminal_state == False:
        print('State: ', state)
        state_encod = np.reshape(state, [1, state_size])
        q_values = model.predict(state_encod)
        action_key = np.argmax(q_values)
        action = index_to_action_mapping[action_key]
        print('Action: ', action)
        next_state, reward, terminal_state = env.step(state, action)
        print('Next_state: ', next_state)
        print('Reward: ', reward)
        print('Terminal_state: ', terminal_state, '\n')
        print('----------------------------')
        episode_reward += reward
        state = deepcopy(next_state)
    print('Episode Reward' + str(episode_reward))
    test_rewards.append(episode_reward)

plt.plot(test_rewards)

Thanks.


texts: Why can't i normalize my data using preprocessing.normalize?

texts:
code:

df['Miles'] = preprocessing.normalize(df['Miles'])

Error msg:

ValueError: Expected 2D array, got 1D array instead:
array=[33.3304678  33.3304678  33.3304678  ...  3.00959404  3.00959404
  3.00959404].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.


texts: How do I print full date in the x axis of the line plot here?

texts:
My code:
import matplotlib.pyplot as plt
plt.style.use('ggplot')
plt.rcParams["figure.figsize"] = [12,6]
def time_series(start, end):
    time_series_df = df[['Date', 'Value']][(df['Date'] >= start) & (df['Date'] <= end)]
    x = time_series_df.Date
    y = time_series_df.Value
    plt.plot(x,y)
    plt.xlabel('Time')
    plt.ylabel('PM2.5 Value')
    plt.title('PM2.5 Time Series')
    return plt.show();

time_series('2014','2019')


texts: How do I recommend items to out of training users based on its recent views?

texts:
I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings.

Since we have a lot of users in system, I had to sample some users to train model to avoid overfitting.

Now how do I construct user embeddings for out of training users. I tried constructing user embeddings by averaging item embeddings for user's items. But when I compared performance of average vector vs original user embeddings, it is not that great.

So how would I generate user embeddings using item matrix and rating matrix?


texts: How to calculate MAE and threshold in a multivariate time series

texts:
I'm trying to understand how to calculate the MAE in my time series and then the thresholds to understand which of my data in the test set are anomalies. I'm following this tutorial, which is based on a univariate time series, and they calculate it in the following way:
# Get train MAE loss.
x_train_pred = model.predict(x_train)
train_mae_loss = np.mean(np.abs(x_train_pred - x_train), axis=1)

I have a dataset structured as well:
device1   device2   device3   ....   device30
 0.20      0.35      0.12              0.56
 1.20      2.10      5.75              0.16
 3.20      9.21      1.94              5.12
 5.20      4.32      0.42              9.56
 ....      ....      ....              ....
 7.20      6.21      0.20              -9.56

This means my threshold will need to care about 30 devices instead of one. I reshaped my train set as (3000, 10, 30), where 3000 are the values, 10 are the TIME STEPS (I did this to prepare the data for the Conv1D) and 30 are the features (the devices). What I thought was to calculate it as well, but it did't work since it gave me a very small threshold:
# Get train MAE loss.
x_train_pred = model.predict(X_train)
train_mae_loss = np.mean(np.mean(np.abs(x_train_pred - X_train), axis=1), axis=1)
# Get reconstruction loss threshold.
threshold = np.max(train_mae_loss)

Any clue on how can I calculate it?


texts: Reducing a page of content to a short paragraph

texts: I remember years ago, Yahoo detailed how they were able to reduce a webpage down to a short paragrah of text succently summarising the content in sentences, as opposed to a list of keywords. What is this called? Are there any open / free code to do this?


texts:
I am trying to implement a recurrent neural network machine translation system, and I am just learning the things.

I am creating a word embedding matrix. In order to do that, I should know my vocabulary size, dimension of the embedding space, and context window size?

What is context window?


texts: How to use a different model to deep neural network with reinforcement learning based on DQN?

texts:
Is it possible to implement a reinforcement learning algorithm without using a deep neural network (DNN) as used in deep reinforcement learning e.g. Deep Q-Network (DQN)?

How can I replace the DNN in the DQN algorithm with another algorithm? Should it be supervised or unsupervised, and what is this called - is it  "un/supervised reinforcement learning" or "un/supervised DQN"?

In the following DQN pseudo-code, if I want to replace a DNN section with another un/supervised algorithm, is it possible? If so, how?


texts: Is it OK to use the testing sample to compare algorithms?

texts:
I'm working on a little project where my dataset have 6k lines and around 300 features, with a simple binary outcome.

Since I'm still learning ML, I want to try all the algorithms I can manage to find and compare the results.

As I've read in tutorials, I split my dataset into a training sample (80%) and a testing sample (20%), and then trained my algorithms on the training sample with cross-validation (5 folds).

My plan is to train all my models this way, and then measure their performance on the testing sample to chose the best algorithm.

Could this cause overfitting? If so, since I cannot compare several models inside model_selection.GridSearchCV, how can I prevent it to overfit?


texts: Terminology - cross-validation, testing and validation set for classification task

texts:
Confusion1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing?

Confusion2) In the following code I have used 10-fold cross-validation for training a Support Vector Machine (SVM). In general a data set will be split into (a) Training set, meas(trainIdx,:) (b) Testing set, meas(testIdx,:) c) Validation set.
In the cross-validation approach I am building the SVM learner by training and validating inside the loop. Based on my understanding, the validation data must be completely different from the training and testing. But, in many online resources it is said that after cross-validation, one must re-train on the entire data set which in this example would be the meas(:,1:end). If so, then the learned model svmModel inside the cross-validation is lost. Have I misunderstood completely wrong?

Can somebody please show what is the next step in the classification once the cross-validation is over?


texts: How to get all data from an ImageDataGenerator?

texts:
I want to get the result of all the processing that the generators performed on my images. For example:
x,y = generator.next()
but for all data, not just one batch (which is what the above does). The purpose is so I can manually calculate the loss using y - yhat, where y is the data that I want from the generator (original data + transforms from the generator), and yhat is the result after passing the data through an autoencoder. I am measuring the reconstruction error of the images, and therefore want the full loss vectors for the images passed through.
I could run generator.next() over and over again, but I am wondering, is there a simple command I could run to do this?


texts: Rearranging data frame from column names to key value pairs

texts:
If I have data like this in R:

> a = data.frame(a = c(1, 2),
               b = c(3, 4))
> a
  a b
1 1 3
2 2 4

but I would like to have it like this:

> b = data.frame(k = c("a", "a", "b", "b"),
+                v = c(1, 2, 3, 4))
> b
  k v
1 a 1
2 a 2
3 b 3
4 b 4

How can I convert the data this way?


texts: What are some best Text Representation techniques in NLP

texts:
I've studied about various text representation techniques like : Bag of Words, N-gram data modelling, Tf-idf, word embedding etc.

I would like to know which among all the techniques are most efficient when it comes to data modelling or representation for a supervised text classification across a large number of categories.

I might have around 40 categories and then around a same number of sub-categories upto 4 levels.


texts: Where are WEKA installed packages stored

texts:
I would like to use the WEKA library in a Java program but I can't seem to find the methods I installed using WEKA's package manager.

Does anyone know where the installed methods are stored?

For clarification, I installed WEKA, installed the extra package using WEKA and can use it with the WEKA GUI. But I can't find the installed package in the .lib file.


texts: LIME visualization outputs padded regions as important - Mel-spectrogram (audio analysis)

texts:
I am encoding audios as Mel-spectrograms and using these Mel-spectrograms as input to my deep learning model (Inception-ResNet V2). The input image is of size 256 X 256, made up of a 128 X 64 spectrogram (128 melbands) along with zero padding. Sample input:

I am using LIME to visualize the important regions of a Mel-spectrogram. In some Mel-spectrograms, LIME is labelling the padded regions as important, like in this image.

So, is this happening because of poor learning or is this a genuine problem with LIME when it encounters padded regions?


texts: Why do all DTW-packages use different step-patterns and distance-methods?

texts:
I already posted it on StackOverflow. I was not sure if this was the right place.
I have several questions about the algorithm DTW. I tried different Python-Packages

DTAI-Distance

DTW-Python
First of all I compared the results (the distance) of these libs. They use different distance-methods. DTAI-Distance uses (x-y)² while DTW-Python uses abs(x-y).

As well DTAI-Distance uses the step-pattern symmetric1 that i don't understand why. On this page it says: "symmetric1 (or White-Neely) is quasi-symmetric, no local constraint, non-normalizable. It is biased in favor of oblique steps." Also here it says:

It cannot be meaningfully normalized by timeseries length. Hence,
longer timeseries have naturally higher distances, in turn making
comparisons impossible.
It favors diagonal steps, therefore it is not robust: two paths
differing for a small local change (eg. horizontal+vertical step
rather than diagonal) have very different costs.

So why do DTAI-Distance uses it? DTW is used to compare timeseries of different length, but if they use symmetric1 as a step-pattern, it seems wrong. So my question: "When to use which step-pattern?" When do i use asymmetric, symmetric1, symmetric2 or some kind of Rabiner-Juang set? Here you can find a similar question, but the answer is still the same. "Just stick with symmetric2". I tried different step-patterns with some data, but i was not able to get it right.
If anyone has a paper, other literature or even an answer I would be very grateful.


texts:
We are using Google BERT for question and answering. We are using vanialla bert-base-uncased as well as squad trained checkpoints.

The answers from BERT are very short and crisp. For example, if we ask describe a chatbot, then it will simply return, takes input from user and replies ...  Though the answer actually is complete paragraph.

Will training BERT on long paragraphs can solve this problem? If yes, any idea from where we can get such a QnA dataset, as it is not possible to create a huge QnA dataset manually.
Is there any other tweak which can be done at some BERT layer, so that it starts understanding a long answer?
Or is there any other framework or system already have solved this kind of problem, by maybe integrating with some other neural network technique, or by using a pipeline of multiple components?


texts: How to find the class name of a new image from the pre-trained model

texts:
I would just like to get the class names of the predictions. I can get the class names on the images that I trained the model. But if I predict an image (say which is not trained but already belongs to the pre-trained model (VGG16) I cannot get its class name).

My scenario:
I used VGG16 pre-trained and added new datasets (like logos, stars, hills etc which are not present in VGG16 (say) )

Now I trained this model in those datasets, when i predicted an image, it gave the correct class-name (logos, stars, hills etc..)

But when I predict an image that was already in VGG16 (say Dog, Cat), it is not giving me the right class name i.e Dog, Cat etc.

I need some help in this scenario.

Is there any way to get the get the class name?


texts:
Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints.

Is this too much data for XGBoost ? I have searched all over the internet for a solution to when the data won't fit into RAM memory to no avail. I read about external memory for xgb but there is no detailed doc. Can someone point me in the right direction please and thank you !


texts: Why do internet companies prefer Java/Python for data scientist job?

texts:
I see a many times in job description for data scientist asking for Python/Java experience and disregard R. Below is a personal email I received from chief data scientist of a company I applied for through linkedin.

  X, Thanks for connecting and expressing interest. You do have good
  Analytics Skills. However,  all our data scientists must have  good
  programming skills in Java/Python as we are a internet/mobile
  organisation and everything we do is online.

While I respect the decision of the chief data scientist, I am unable to get a clear picture as to what are the tasks that Python can do that R cannot do. Can anyone care to elaborate? I am actually keen to learn Python/Java, provided I get a bit more detail.

Edit: I found an interesting discussion on Quora.
Why is Python a language of choice for data scientists?

Edit2: Blog from Udacity on Languages and Libraries for Machine Learning


texts: Recommendations for storing time series data

texts:
As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, but it's made me wonder whether there are best practices out there for this sort of task.

I'll describe what I've got, and maybe that will help provide some recommendations. So, I have an experiment that requires subjects to use their vision and move their body to complete a task. Each task is one trial, and each subject performs multiple trials to complete the experiment. During a trial I record the movement and the eye tracker (~200 channels) at regularly sampled time points (~100Hz). I store these in a CSV file (one file per trial), with one row per time point, and one column per variable (e.g., left-fingertip-x, left-fingertip-y, left-fingertip-z, etc. for the mocap, and left-eye-x, left-eye-y for the eyes).

Associated with each trial is some metadata such as the experimental condition of the trial (e.g., how fast a target in the trial is moving, say). I store these values in the CSV filename itself, using a "key=value" sort of syntax.

While this works well enough for my purposes, it's really ad-hoc! I'd like to get a sense of whether other people have solved problems like this, and, if so, how?


texts: Regex-style pattern-matching for time series

texts:
This is more of a "what technology/library would you use for this?" question than anything else.

I have categorical time series data, and need to match cases in these time series to known patterns. For example, State A, followed by State B within six months, followed by multiple periods of State A, followed by an optional period of State B, followed by State C.

It's quite hard to interrogate e.g. an SQL database or pandas dataframe for sequences of rows that match a pattern like this. How you'd do this "in theory" is build a pattern-matching finite state machine, but I didn't know a relevant library for one. Then I realised I've been using pattern-matching finite state machines for years in the form of regular expressions.

So, what I've been doing is converting my time series to strings, defining my pattern as a regex, and matching them that way. It works extraordinarily well. In spite of this, it feels hacky. Surely there should be some proper way to do this.

Is there? What technology/data structure/library would let me define finite automata for time series and match them based on that definition?


texts: Why is it necessary to clean/preprocess data if the unseen real data is bound to consist missing values, outliers etc.?

texts: Can it really be called Generalization if we remove/modify data points to suit our model?


texts: How does the Nearest Centroid method work?

texts:
I have read this Wikipedia article.

But, the idea is still very fuzzy to me.

Suppose, k=5.

Then, we have,

$X_5 = \{A, B, C, D, E\}$

$Y_2 = \{Triangle, Square\}$

$R_5 = \{9, 8, 5, 1, 4 \}$ (just assumed)

Now, $\mu_{Triangle} = \frac{5}{2} = 2.50$

and, $\mu_{Square} = \frac{22}{3} = 7.33$

Since, $\mu_{Triangle} < \mu_{Square}$, $class(?) == Triangle$.

Am I correct?


texts:
I am looking for a definition of Data Scout.

what is the difference between Data Mining and Data Scout?


texts: How do I compare traditional classifiers performance with my proposed method?

texts:
I want to compare my proposed method with traditional machine learning classifiers like Multilayer Perceptron(MLP) and SVM to check the classification accuracy.How do I compare different classifiers?. I want to draw a comparison between these three terms

1).Proposed method (Based on DNN)
2).Shallow neural networks
3).SVM


texts:
I am using the Keras library (with Python 3.6) to create a neural network.

My network maintains a constant overall maximum accuracy of 62.5%, over 16 training samples.

In what ways can I increase this accuracy?

Should I increase the number of training samples, or will restricting some of the data in the training samples help? Or something else that I might not know of?

Any help is greatly appreciated.

Here are the layers of my neural network:

# Build neural network
# Neural net with multiple layers
model = Sequential()

model.add(Dense(32, input_dim=17, init='uniform', activation='sigmoid'))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(32, init='uniform', activation='relu'))
model.add(Dense(16, init='uniform', activation='sigmoid'))
model.add(Dense(4, init='uniform', activation='sigmoid'))
model.add(Dense(1, init='uniform', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit model
history = model.fit(X, Y, validation_split=0.46, nb_epoch=150, batch_size=3)


texts: How to replace NA Data using a regression?

texts:
My dataset contains some stocks and some bitcoin titles.
I would like to replace NA data (normal stocks are not traded on sat. and sun.) using a regression on the stock itself.
How can i do this?


texts: How to creat a plot for the accuracy of a model

texts:
Iam pretty new to the whole topic so please dont be harsh. I know these may be simple questions but everybody has to start somewhere ^^

So I created (or more copied) my first little Model which predicts sons heights based on their fathers.

#Father Data
X=data['Father'].values[:,None]
X.shape

#According sons data
y=data.iloc[:,1].values
y.shape

#Spliting the data into test and train data
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)

#Doing a linear regression
lm=LinearRegression()
lm.fit(X_train,y_train)

# save the model to disk
filename = 'Father_Son_Height_Model.pckl'
pickle.dump(lm, open(filename, 'wb'))

#Predicting the height of Sons
y_test=lm.predict(X_test)
print(y_test)

Now I wanted to create a plot that displays the accuracy of my model or how "good" it is.
Something like it is done here:
https://machinelearningmastery.com/how-to-calculate-precision-recall-f1-and-more-for-deep-learning-models/

But i cant quiet get it to work. This here Sems like I would work but what should I store in "model_history"?

plt.subplot(212)
plt.title('Accuracy')
plt.plot(model_history.history['acc'], label='train')
plt.plot(model_history.history['val_acc'], label='test')
plt.legend()
plt.show()

A easy to adapt tutorial-Link would be a geat help already.
Keras seems to be a thing but I would want to avoid yet another library if possible and sensible.


texts:
I try to predict the position of a specific point (crest) in a 1D signal (elevation profile). Until now, I computed gradient at every point of my signal and combined that with additional features or heuristics to find approximate position of the expected output (position of the crest).
But there are some limits of this approach and I've found that ML techniques, and especially Random Forest classifiers could perform well in this kind of situation.
I would like to train my RF to find the most probable point (point_index) being the "output" based on a profile input.
Yet, I only found examples of training RF models with 1D inputs (like a time series). In my case, I have 2D input data (one signal is composed of N points with 2 features associated to each point) like the following dataframe :
   profile_index  point_index         z             z'        crest
0              0            1 -0.885429             0          false
1              0            2 -0.820151          0.02          false
2              0            3 -0.729671          -0.1           true
3              0            4 -0.649332           0.1          false
4              1            1 -0.692186             0          false
5              1            2 -0.885429           0.1           true
6              1            3 -0.820151         -0.05          false
3              1            4 -0.649332           0.2          false

I can map my data to split the dataframe for every profile, and get the output point_index as a feature, but how do I manage the fact that 2 of my features are arrays ?
Edit: here is another representation for my data
   profile_index               points_z         points_z_prime    crest_index
               0     [-0.05, ..., 2.36]        [0, ..., -0.01]            150
               1     [-0.02, ..., 4.41]        [0, ..., -0.02]            162

(this is probably irrelevant regarding the method, but I work with Python and scikit-learn)


texts: What is "$D$ equal to $\frac{1}{2}$ everywhere" in original GAN paper

texts: In generative adverserial networks paper by Ian Goolodfellow, Y Bengio et al. the authors mention that "$D$ equal to $\frac{1}{2}$ everywhere" what does it means actually? here is the paper


texts: How to train model to predict 1 value from multiple input samples

texts:
Very new in ML...

Actual problem is more complex, but I'll give it shorter.

Train data is a collection of samples, describing features of some items. There are many samples for each item. So ML model is implied to predict a few target variables, but one time for each item:

It appears, that I can't find in sklearn or a similar kit a way to fit model with this type of train-input/train-output. Is there any way to do this without hand-implementing whole pipeline?

I've considered this answer: How to predict based on multiple samples?

But I'm not sure it's applicable, since there are many-many samples for each item

Edit 1:
Final model should get a few new samples for completely new item and predict same target variables


texts: Sports prediction using machine learning

texts: I am trying to predict soccer scores using past results. The dataset I have only consists of the home team, the away team, goals scored by the home team and goals scored by the away team in each match. How can I use the limited data on my dataset to model this using a random forest algorithm and the naive bayes algorithm?


texts:
I have a database (PostgresSQL) which looks like this:

""results"":
    {""WEBS"":
        {""WEBS_4"": [{""ip"": ""XXXXX"", ""value"": ""YYYY""}]}}

I would like to access the information of ip.

Then I write the following query in R but doesn't give me any info

dbGetQuery(conexion,"SELECT
              data->'results'->'WEBS'->'WEBS_4' ->'ip' as test
                          FROM results;")

And when I write this query

dbGetQuery(conexion,"SELECT                   data->'results'->'WEBS'->'WEBS_4' as test
                          FROM results;")

I get a result like this [{""ip"": ""XXXXX"", ""value"": ""YYYY""}], that is not very useful in R.

Some other parts of my database are highly nested, so I would like to access those parts which are with the [] signs.

Thanks a lot!


texts: Is this attribute numeric or categorical (ordinal)? Help!

texts:
So I have this dataset I need to perform several techniques on as part of a data mining/machine learning project of some sort in PYTHON. There are a couple of features however, that have me very worried as I don't know whether I whould handle them as Categorical (ordinal to be precise) or as numeric.

The first feature takes integer values from 0 to 9 and it's supposed to represent population percentage of a certain religion (so 0 is 0%, 1 is 1-10%, 9 is 100%)
The second feature represents contibution (money) and also takes values from 0-9 depending on how big the contribution is (so 0 is.. well, 0 , 1 is 1-49 , 9 is 20000+).

How should I handle these features? Are they interval data? Should I one-hot-encode them into dummies or should I leave them as they are now?
I am going to have to perform some clustering and classification techniques on the dataset.
Thank you in advance!


texts:
I know that there are methods that help in selecting features such as Matual Info, and Info Gain, etc.
But for datasets with thousands of records and thousands of features it is time consuming to train the model in BOW and TFIDF to decide which method is better.
is there a way to decide which method to choose without the need to spend all this time?


texts:
Assume that I have this data set (a taxonomy of parent-child features)
id1 label1
     parentid features
     -------- --------
     id1. features1
     id1. features2
     id1. features N1

id2 label2
     parentid features
     -------- --------
     id2. features1
     id2. features2
     id2. features N2

and I want to develop a classifier of parent using the features of child (eg. id2, child, label2)
available solution:

I can use RNN m - 2 - 1
Graph Neural Network (but doesn't converge)
...?

can you give me an idea/paper of a structured classification.
Please notice: N1 and N2 are different, otherwise I can make a flatten table
eg.
label1 features1 ... featuresM


texts: Using Trainable=True in Keras Embedding obtained better performance

texts:
It is suggested by the author of Keras [1] to use Trainable=False when using the embedding layer in Keras to prevent the weights from being updated during training. But in my experience, I always got better performance (lower error in regression) when setting Trainable=True in text processing.

My question, can I claim my result or I must use Trainable=False?
Do I make a cheat when using Trainable=True?

According to my understanding, it just a choice to use (or not to use) previous information to update the weight, so it is allowed to use Trainable=True.

[1] https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html


texts: How to prepare/augment images for neural network?

texts:
I would like to use a neural network for image classification.  I'll start with pre-trained CaffeNet and train it for my application.

How should I prepare the input images?

In this case, all the images are of the same object but with variations (think: quality control).  They are at somewhat different scales/resolutions/distances/lighting conditions (and in many cases I don't know the scale).  Also, in each image there is an area (known) around the object of interest that should be ignored by the network.

I could (for example) crop the center of each image, which is guaranteed to contain a portion of the object of interest and none of the ignored area; but that seems like it would throw away information, and also the results wouldn't be really the same scale (maybe 1.5x variation).

Dataset augmentation

I've heard of creating more training data by random crop/mirror/etc, is there a standard method for this?  Any results on how much improvement it produces to classifier accuracy?


texts: Multi-class classification with extremely small dataset

texts:
I am working on a text classification task that contains 216 labeled paragraphs. The distribution of tags is as follows:
{0: 17, 1: 15, 2: 16, 3: 9, 4: 10, 5: 18, 6: 24, 7: 9, 8: 33, 9: 38, 10: 27}.
The keys are representing the classes, and the value is the number of samples.
Here's my questions:
Q1. Is there any way to train a classifier?
Q2. If we want an 11-class text classifier how many samples do we need?


texts: Projecting 3D Chemical Data Onto a 2D Plane in Motion

texts: I'm trying to model the rotation of two hydrogen atoms about a carbon atom. Say I have a conceptual wheel on an axle that is attached to my car. The axle is described by two points in 3D space, as well as vector (x, y, z) pointing outward, toward the wheel. Now, there is some 2D plane the is perpendicular/orthogonal to that vector and touching the outermost point as well. The overall motion of the axle and 2D plane can be in all three dimensions, just like if I was driving a car around. Now imagine on the edge of the wheel I have a sensor, which collects positional data as I spin the wheel and drive my car around. The sensor data I collect approximately follows the 2D plane that is orthogonal to the axle vector. Given I have time series data that describes the axle points/vector and wheel sensor readings, how do I find the plane perpendicular to the axle vector, and how do I project the 3D sensor data onto that 2D plane?


texts: Searching for a dataset that targets difficult words

texts:
I am trying to find a dataset in which dataset targets words that are difficult. I understand there would be different levels of difficulty for each individual , but if we considered an average individual, I want to detect the difficult words that would be present in a sentence.
Example:
Yes, may be today's Britains are not responsible for some of these reparations but the same speakers have pointed with pride to their foreign aid - you are not responsible for the people starving in Somalia but you give them aid surely the principle of reparation for what is the wrongs that have done cannot be denied.
IN the above sentence, the model should detect the word - reparations successfully.


texts: Does ensemble (bagging, boosting, stacking, etc) always at least increase performance?

texts:
Ensembling is getting more and more popular. I understand that there are in general three big fields of ensembling, bagging, boosting and stacking.

My question is that does the ensembling always at least increase the performance in practice? I guess mathematically, it is not true. I am jsut asking a real life situation.

For example, I could train 10 base learner, and then stack them with another learner, which is at 2nd level. Does this 2nd-level learner always outperform the best of the base learners in practice?


texts: What are some key strengths of BERT over ELMO/ULMFiT?

texts: I see BERT family is being used as benchmark everywhere for NLP tasks. What are some key strengths of BERT over models like ELMO or ULMFiT?


texts:
I am playing with a credit fraud detection dataset at Kaggle. An imbalanced dataset with about 0.1% of fraud transaction. The features are 28 PCs out from a PCA exercise done by the data publisher + time & txn amount and a class variable of 0/1 for legit/fraud txn.

From my brief understanding, collinearity should have been dealt with during PCA. However, I found that PCs are still correlated among the fraud cases (if you break the dataset up into legit/fraud cases). What should be a good approach to minimise that effect for fraud detection using a Naive Bayes classifier?

Another thing is that I have been taught in DS101 to deal with outliers. However, I don't seem to think removing outliers is a wise choice given that fraud can be an outlier by itself. What are some common approach to deal with outliers while not removing them?

P.S. I am fairly new to Data Science so any good directions on the above topics would be welcome. It is just not as clear cut as I have seen in the introductory text.

Thanks.


texts: Is Label Encoding with arbitrary numbers ever useful at all?

texts:
From what I read online, there seems to be some confusion regarding the taxonomy and the terms used, so to avoid misunderstanding I'm going to define them here:
Label Encoding - encoding a nominal variable with arbitrary numeric labels.
Ordinal Encoding - encoding an ordinal variable with numeric labels arranged in a specific order.
The course on Machine Learning I'm currently taking compares One-Hot Encoding with Ordinal Encoding. However, during my research online I came to realize that "Ordinal Encoding" is actually a misnomer, and what that course actually demonstrates is called "Label Encoding". Ordinal Encoding is supposed to pertain strictly to ordinal variables, and the dataset in question didn't even have any ordinal variables.
Where did that misnomer come from? Turns out that it comes from the scikit-learn library that has LabelEncoder and OrdinalEncoder classes. The thing is, OrdinalEncoder class actually does not perform Ordinal Encoding by default. To make it ordinal, you have to specify the order in the 'categories' parameter (and its usage is extremely not-user-friendly - dictionary mapping by pandas can do the same way easier). If you don't, OrdinalEncoder will assign labels alphabetically, just like LabelEncoder does. So the real difference between these two classes is that one encodes only a single column, while the other encodes two or more columns at a time. Perhaps it would be better and much less confusing if these classes were called "LabelEncoder1D" and "LabelEncoder2D".
So that's where mistakenly calling Label Encoding "Ordinal Encoding" is coming from. But getting back to the question, the course I'm taking advocates the usage of (what I learned to be) Label Encoding for tree-based algorithms, because One-Hot encoding performs much worse for trees (which is ture). However, from what I read online, it appears that other Machine Learning platforms, such as R or H2O, are capable of processing nominal variables for trees without any kind of encoding at all, and the requirement to encode everything into numeric form seems to be exclusively scikit-learn's problem. And there's conflicting information as to whether trees perform better with Label Encoding - my course, as well as some responses online, advocate for its usage, but my intuition, as well as some other responses online, seem to indicate that scikit-learn trees will not be able to distinguish these labels as categories, and will mistakenly assume that they are continuous values on a meaningful scale instead. So they recommend using One-Hot Encoding even for trees as the only option despite it being sub-optimal.
So my questions are 1) is it true that Label Encoding will be misinterpreted as a numeric scale by scikit-learn trees? 2) if so, are there any situations at all where arbitrary Label Encoding can be useful? Or does this technique has no use at all unless the variable is ordinal, and a specific labeling order is given?
P.S: I'm asking because my course has a whole lesson dedicated to teaching students "Ordinal" Encoding. At first I wanted to suggest them to rename it to "Label Encoding", but now I suspect that that whole lesson is best removed altogether to avoid teaching students bad practices.


texts: Changing the value of a column based on the condition on another column

texts:
I have a table whose header looks like this:
    complaint_type  borough street_name incident_zip    latitude    longitude

I want to check if the "incident_zip" column of each row is in a specific list of zip codes and change the "borough" accordingly. There is a large amount of data and i cannot find any better code to do this. I am using python 3.6.


texts: cross validation on whole data set or training data?

texts: I am always having cross validation score smaller then the training score and I am performing cross validation on just training data is that normal thing ? Kfold = 5


texts: neural networks error function: is global minimum desirable?

texts:
In "Elements of statistical learning" page 395 the authors state that, relative to R(θ), the regression/classification error function in a neural network such as a multi layer perceptron:

  Typically we don’t want the global minimizer of R(θ), as this is
  likely to be an overfit solution. Instead some regularization is
  needed: this is achieved directly through a penalty term, or
  indirectly by early stopping. Details are given in the next section.

However in Backpropagation, when momentum regularization is described the figure 9 shows how the back propagation process should steer the error function towards its global minimum, avoiding any local minimum found along the way.

How come these 2 reasonings are compatible? The only thing I can think of is that we want to get the global minimum for each coefficient error (as computed during the backprop process) while not adjusting for the "global" error function. Am I interpreting this correctly?


texts: Keras/Tensorflow: model.predict() returns a list. How do I match the output with my class names?

texts:
I have a CNN built in Keras. I have saved it and am now using the model.predict() function to make predictions from it. Whenever I run the following code,
def prediction(path):
    import keras
    from keras.preprocessing.image import load_img, img_to_array
    from keras.models import load_model
    import PIL
    import numpy as np

    img = load_img(path)
    img = img.resize((224, 224))
    img = img_to_array(img)

    img = img.reshape( -1,224, 224,3)

    model = load_model('model1.h5')
    pred = model.predict(img)

    return pred

print(prediction('/path/to/image/')

I get an output like this:
[[7.578206e-37 1.000000e+00 0.000000e+00 0.000000e+00]]
I am doing transfer learning using resnet50 with imagenet weights and here is the model.summary().
I have 4 classes. How do I find out where each prediction belongs?
I have looked here as well but it doesn't seem to help me.
Thanks


texts: Deep fully connected NN with vanishing gradients

texts:
I am making NN for choosing best bets possible for football matches. And I tried to make network quite deep (12 hidden layers with BN between them and ReLu as activation function) but it resulted in vanishing gradients problem.
Then I made it shallow (2-3 layers in same manner) and it resulted in great performance in train set but poor in validation set. My hypothesis is that in this configuration it could memorize all the possibilities (I have about 14 k of examples in train set).
Finally I made it in between this two - 8 hidden layers. And it resulted in quite nice performance both in train and val set. But i'm still a little worried about vanishing gradients and that it might be just a coincidence.
So I have few question in reference to what I've just written:

When should you be worried about vanishing gradients? I mean I can
inspect some gradients distribution in tensorboard but this doesn't
give me 100% sure answer.
Could using skip connections help?
Is there any rule how much layers is too much?


texts: xgboost: Is there a way to perform regression on rates/percentages data?

texts:
I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific subsets of the data in training itself, but I am stumped as how to perform regression on the data I have.

Normal OLS regression will produce outputs that will be over and under the [0,1] range if you do not change the likelihood to be the Beta distribution, or something else bounded in the same range, but will xgboost suffer the same mistake?

Any advice on trying to get this to work would be greatly appreciated by me.


texts: Reshape features in rows intro columns python?

texts:
My features are arranged in rows, Can anyone tell me please how can I transform them into columns with theirs corresponding times and values??
I mean each features has a set of corresponding times and values ..
a view example of   dataset is :

Feature                      TIME                     Value
    YAM_B55P113BDPMN  2015-06-14T04:05:18.000Z            0
    YAM_B55P113BDPMN  2015-06-14T04:06:18.000Z          Bad
    YAM_B55P113BDPMN  2015-06-14T04:10:08.000Z            0
    YAM_B55P113BDPMN  2015-06-14T04:13:18.000Z          Bad
    YAM_B55P113BDPMN  2015-06-14T04:26:28.000Z          Bad
    YAM_B55P113BDPMN  2015-06-14T04:31:48.000Z            0
    YAM_B55P113BDPMN  2015-06-14T05:07:28.0
        .
.......
   YAM_LIK13.SET.TTCI  2017-05-28T03:36:22.000Z          42
  YAM_LIK13.SET.TTCI  2017-05-28T07:23:31.000Z           42
  YAM_LIK13.SET.TTCI  2017-05-28T11:30:12.000Z           42
  YAM_LIK13.SET.TTCI  2017-05-29T03:16:42.000Z           42
  YAM_LIK13.SET.TTCI  2017-05-29T06:56:12.000Z           42


texts: how to calculate quadratic features in computer vision Neural Network

texts:
I am recently watching some tutorials for deep learning from Dr Andrew Ng on Youtube. Link is hereThe Youtube video

There is a concept of number of features in convolutional neural network in TensorFlow's tutorial.

I don't quite understand why the feature is 32 or 64 here in conv layer1 or layer2?

Then I came to the video, there is also the concept of Quadratic features. It is calculated as 3 million however. But how is it calculated?

Are the two features related in concept?


texts:
I am planning to create a speech recognition network that recognize few words (voice commands) and came across Speech Commands dataset from google. Apart from available dataset I am planning to add few more words like "move", "save" etc, which are not part of the google's dataset.

Is there a way or any other source where i can create/find the dataset for the words ?

As many of the online dataset are available for sentences and speech transcripts i am thinking of writing a scripts that can go through the available transcripts and find the location of the desired word and physically cropping the audio and then padding it to make one second audio file. But this seems very very arduous process and few words like the name of my robot will not be available online.

Can you guys suggest a way ?


texts:
I have directed graphs with each vertex has some label in "class01_01.gml" format. There are multiple such classes. I need to do multi class classification on these files using deep learning.

Note that each graph in a particular class has different set of vertices.

Is anyone familiar how to apply deep learning in this context?

PS: I'm not that familiar in applying deep learning on graphs.


texts: How to deal with class imbalance problem in natural language processing?

texts:
I am doing a NLP binary classification task, using Bert + softmax layer on top of it. The network uses cross-entropy loss.
When the ratio of positive class to negative class is 1:1 or 1:2, the model performs well on correctly classifying both classes(accuracy for each class is around 0.92).
When the ratio is 1:3 to 1:10, the model performs poorly as expected. When the ratio is 1:10, the model has a 0.98 accuracy on correctly classifying negative class instances, but only has a 0.80 accuracy on correctly classifying positive class instances.
The behavior is as expected as the model turns to classify most/all instances toward negative class since the ratio of positive class to negative class is 1:10.
I just want to ask what's the recommended way for handling this kind of class imbalance problem in natural language processing specifically?
I saw someone suggests to change loss function, or perform up/down sampling, but most of them are targetting computer vision class imbalance problem.


texts: Question about the simple example for batch normalization given in "deep learning" book

texts:
In the section about batch normalization of Deep Learning book by Ian Goodfellow (chapter link) there is the follwing text:

  As  example, suppose we have a deep neural network that has only one
  unit per layerand does not use an activation function at each hidden
  layer: $y=x w_1 w_2 w_3 \ldots w_l$. Here, $w_i$ provides the weight used by
  layer $i$. The output of layer $i$ is  $h_i=h_{i−1} wi$. The output $y$ is a
  linear function of the input $x$, but a nonlinear function of the weights
  $w_i$.

Why y is nonlinear with respect to w_i?


texts: Terminology in machine learning: exogenous features vs external features

texts:
I am currently writing a scientific paper and do not know whether to call some of my input features of my neural network either external or exogenous.
My neural network receives as input features like the outside temperature, which are completely independent of the mapped mathematical function.
Are these independent features called exogenous or external? When is which term used?


texts:
I have some questions about Custom Writable objects in Hadoop. First of all, the private instance variables of the Writable object have to be Writable?

How does Hadoop handle TreeSet<IntWritable> differently from ArrayWritable<IntWritable> in org.apache.giraph.utils (Class ArrayWritable<T extends org.apache.hadoop.io.Writable)?

I have the following variables:

private TreeSet<IntWritable> vertexList;
private IntWritable shortestPathCounter;
private IntWritable totalShortestPathCounter;
private DoubleWritable bCentrality;

and the following method for serializing. Are write and readFields method correctly handled this way?

public void readFields(DataInput in) throws IOException {
    for(IntWritable i: vertexList)
        i.readFields(in);
    shortestPathCounter.readFields(in);
    totalShortestPathCounter.readFields(in);
    bCentrality.readFields(in);
}

public void write(DataOutput out) throws IOException {
    for(IntWritable i: vertexList)
        i.write(out);
    shortestPathCounter.write(out);
    totalShortestPathCounter.write(out);
    bCentrality.write(out);
}


texts: Protein interaction prediction- how to input this data structure

texts: I know the basics of machine learning and have quite an experience with time series data or data fed in a tabular format. But in the picture, the data is arranged as a graph. Is there a way to input the graph into a ML tool such as Artificial Neural Network or any other? I don't know if there is a theory for handling such data structure. The task is to recreate the graph from the output of the ML algorithm after training. So, whatever input I get, the output should be the same as the input -- quite similar to an auto-associative memory. Can somebody please help?


texts:
I am trying to start the learning of a cnn network which has 72 input and one output being a vector of length 24 stating the a class for each third input 72/24 = 3. There are 145 classes.

this is how i've designed the network currently passed my data:

print "After test_output/input"

print "Length:"
print len(data_train_input)
print len(data_train_output)

print len(data_test_input)
print len(data_test_output)

print "Type;"
print type(data_train_input[0])
print type(data_train_output[0])

print "Size [0]"
print data_train_input[0].shape
print data_train_output[0].shape

list_of_input = [Input(shape = (78,3)) for i in range(72)]
list_of_conv_output = []
list_of_max_out = []
for i in range(72):
    list_of_conv_output.append(Conv1D(filters = 32 , kernel_size = 6 , padding = "same", activation = 'relu')(list_of_input[i]))
    list_of_max_out.append(MaxPooling1D(pool_size=3)(list_of_conv_output[i]))

merge = keras.layers.concatenate(list_of_max_out)
reshape = Reshape((-1,))(merge)

dense1 = Dense(500, activation = 'relu')(reshape)
dense2 = Dense(250,activation = 'relu')(dense1)
dense3 = Dense(1 ,activation = 'softmax')(dense2)

model = Model(inputs = list_of_input , outputs = dense3)
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam" , metrics = [metrics.sparse_categorical_accuracy])

reduce_lr=ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=3, verbose=1, mode='auto', epsilon=0.01, cooldown=0, min_lr=0.000000000000000000001)
stop  = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto')

print "Train!"
list_train_input = []
list_test_input = []

for i in range(len(data_train_input)):
    list_train_input.append(data_train_input[i])

for i in range(len(data_test_input)):
    list_test_input.append(data_test_input[i])

hist_current = model.fit(x = [np.array(list_train_input[i]) for i in range(72)],
                    y = np.array(data_train_output),
                    shuffle=False,
                    validation_data=([np.array(list_test_input[i]) for i in range(72)], np.array(data_test_output)),
                    validation_split=0.1,
                    epochs=150000,
                    verbose=1,
                    callbacks=[reduce_lr,stop])

Which generates this output:

After test_output/input
Length:
9436
9417
1017
1035
Type;
<type 'numpy.ndarray'>
<type 'numpy.ndarray'>
Size [0]
(72, 78, 3)
(24,)
Train!
Traceback (most recent call last):
  File "keras_cnn_phoneme_classification.py", line 382, in <module>
    model(train_input_data_interweawed_normalized, output_train, test_input_data_interweawed_normalized, output_test, test_name_interweawed_normalized)
  File "keras_cnn_phoneme_classification.py", line 361, in model
    callbacks=[reduce_lr,stop])
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1405, in fit
    batch_size=batch_size)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1299, in _standardize_user_data
    exception_prefix='model target')
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 133, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking model target: expected dense_3 to have shape (None, 1) but got array with shape (9417, 24)

Why am i getting this error?
I tried changing the output size of the dense3 to many  different?  but is is it expecting?


texts: Data science projects explained step by step?

texts: I am looking for a website or book where several practical examples are given step by step, explaining how they choose the relevant features, the model selection procedure, etc...


texts:
I am trying to create a neural network using time series as input, in order to train it based on the type of each series.
I read that using RNNs you can split the input into batches and use every point of the time series into individual neurons and eventually train the network.

What I am trying to do though is use multiple time series as an input. So for example you might receive input from two sensors. (So two time series), but I want to use both of them in order to get a final result.

Also I am not trying to predict future values of the time series, I am trying to a get a classification based on all of them.

How should I approach this problem?

Is there a way to use multiple time series as an input to an RNN?
Should I try to aggregate the time series into one?
Or should i just use two different neural networks? And if this last approach is correct, if the number of time series increases wouldn't that be too computer intensive?