texts: What is the best approach for specified optical character recognition?

texts:
I have a quite understandable request of extracting information (invoice number, invoice data, due date, total etc.) from scanned invoices (the digital format is image, not PDF), preferably in Python. The good thing is that the necessary information is more or less certain to exist on the page, and the (regexp-like) textual format of these is also tend to be consequent. The downside on the other hand is that the layout of the invoices are very diverse.

I have played with the following possible approaches:

Use character recognition to extract pure text and later try to
puzzle with the fragments. This method has some considerable
problems: the quality of the OCR (at least of the implemented one in
tesseract library) are quite mediocre and the output is hopelessly
unstructured (practically a big pile of words), it is very difficult
to come out any regexp or other rule even for regular phrases.
My other approach would be to apply some kind of deep learning either to
the raw image itself or the text pile where we leave the heavy
lifting to the network, but in this case I'm not sure what is
supposed to be the output? Is it some kind of a sequence to sequence
mapping?

Very unusual task, indeed.


texts: Why Decision trees performs better than logistic regression

texts: I'm working on a machine learning project, a classification of (100 x 100) Images (every pixel contains 0 or 255), my training set contains 10000 examples (which I split into 2 parts 80% training/20% for testing). When I applied Logistic regression algorithm I had 22 % of accuracy and when I applied a decision trees algorithm I had 55 % of accuracy, but I can't understand why the second algorithm gives a better result.


texts: NLP in Turkish with a DV that is a continuous number

texts:
I know how to do NLP in R. Specifically, I know how to train a set of reviews on ratings and then predict ratings for a test set of reviews when the ratings are two-dimensional, i.e. 0 or 1. How can I do the same in Turkish and when my ratings are 5-dimensional, i.e. 1, 2, 3, 4, 5?
Thanks for your helps


texts: How to use TFIDF vectors with multinomial naive bayes?

texts:
Say we have used the TFIDF transform to encode documents into continuous-valued features.

How would we now use this as input to a Naive Bayes classifier?

Bernoulli naive-bayes is out, because our features aren't binary anymore.
Seems like we can't use Multinomial naive-bayes either, because the values are continuous rather than categorical.

As an alternative, would it be appropriate to use gaussian naive bayes instead? Are TFIDF vectors likely to hold up well under the gaussian-distribution assumption?

The sci-kit learn documentation for MultionomialNB suggests the following:

  The multinomial Naive Bayes classifier is suitable for classification
  with discrete features (e.g., word counts for text classification).
  The multinomial distribution normally requires integer feature counts.
  However, in practice, fractional counts such as tf-idf may also work.

Isn't it fundamentally impossible to use fractional values for MultinomialNB?
As I understand it, the likelihood function itself assumes that we are dealing with discrete-counts:

(From Wikipedia):

${\displaystyle p(\mathbf {x} \mid C_{k})={\frac {(\sum _{i}x_{i})!}{\prod _{i}x_{i}!}}\prod _{i}{p_{ki}}^{x_{i}}}$

How would TFIDF values even work with this formula, since the $x_i$ values are all required to be discrete counts?


texts: How does the equation "dW = - (2 * (X^T ).dot(Y - Y_hat)) / m" comes in Linear Regression (using Matrix + Gradient Descent)?

texts:
I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per Iteration:
y_hat = X.dot(W) + b

dW = - (2 * (X^T ).dot(Y - Y_hat)) / m # how does the minus and matrix multiplications are used instead of Summation?
db = - (2 * np.sum(Y - Y_hat)) / m  # np is numpy

W = W - lr * dW  # update weights
b = b - lr * db

I know from the code is that dW is derivative of the weight matrix per iteration, X^T is the Transpose of the X features, Y is original values of Y , Y_hat are predicted values using the formula X.dot(W)+b.
What I want to know is that dW = - (2 * (X^T ).dot(Y - Y_hat)) / m. Even with the MSE loss, as given in this link in the equation 1.4, it should be something else.
Can someone please elaborate how are the values of dW,db are calculated here?
Whole Python Code is giveb here for Linear Regression


texts: Kalman filter for time series prediction

texts:
I have the information about the behaviour of 400 users across period of 1 months (30 days).
Across those 30 days I measure 4 different information (let's call it A,B,C and D), hence I have a total of 4 time series.
My goal is to predict, starting from day 5, the future values of A, by knowing the past values of A, B, C and D.
So:

A(5) = f(A(1,2,3,4), B(1,2,3,4), C(1,2,3,4), D(1,2,3,4))

Ideally, I'd like to estimate the whole future values, from day 5 to day 30.

I tried to use LSTM by using as input the past series, to predict the future trends.
However, I am not getting great results, and I am hoping I can use a simpler approach.
I came across with the concept of Kalman Filter, where the value at time step k can be estimated by using the past values.
Here there are some nice examples , however I have no idea where to start from.
Would it fit with my problem, if yes, how could I fit my past values into the equation for the estimation of A(k)?
Any suggestion is appreciated.


texts: Manipulating multi-indices for a pandas dataframe

texts:
I have a pandas dataframe with multi-index. I have couple of questions on this.

The indices are week numbers (38 to 42) and for each week, day of the week (DOW). So it looks like

The problem is the 2nd level index, that is, DofWeek, is automatically sorted in lexicographic ordering. But I want to have the usual weekdays ordering, i,e for week 39, I want it to appear as Monday, Tuesday, Wednesday, Thoursday, Friday, Saturday and Sunday and NOT in the order they appear. So when I graph it, it appears this way:

I want it to appear in the usual Monday, Tuesday, Wednesday etc. I have tried reindexing, but it didn't work. This is my first question.

My second question is regarding querying such dataframes. For example when I do dataframe.query('DofWeek == 'Saturday'), it is throwing an error saying Saturday is not defined. However, if I swap levels and then query, it works. For example dataframe.swaplevel().query('weekNum == 39'), it works perfectly. As far as I know, the multi-indices should be sorted in order for certain methods to work. But in my case, they are. Then why isn't it working.

Any help would be hugely appreciated.

Thanks.


texts: I'm working on feature weighting techniques (chi-square, relief..) for classification tasks using Weka. Can I add these weights to the dataset's attributes? If yes, how? Do the classification algorithms in Weka make use of the feature weights?


texts:
I am learning about "Image Denoising using Autoencoders". So, now I want to build and train a model. Hence, when I read into how Nvidia generated the dataset, I came across: We used about 1000 different scenes and created a series of 16 progressive images for each scene. To train the denoiser, images were rendered from the scene data at 1 sample per pixel, then 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, and 131072 samples per pixel.

I was trying to understand-

1) what is meant by rendering images at n samples per pixel?

2) How to do this in python to generate the dataset?

I have read some articles regarding this but could not form a confident opinion from a Data Science perspective.

https://area.autodesk.com/tutorials/what-is-sampling/

Any leads would be much appreciated! Thanks


texts: Summary statistics by category using Python

texts:
I have a datset with Scores and Categories and I would like to calculate the summary statistics for each of these categories. The data look something like this:

Category    Score
AAAA        1
AAAA        3
AAAA        1
BBBB        1
BBBB        100
BBBB        159
CCCC        -10
CCCC        9

What I would then like would be something like this

Category    Count   Mean   Std   Min   25%   50%   75%   Max
AAAA
AAAA
AAAA
BBBB
BBBB
BBBB
CCCC
CCCC

I have been looking at using pandas with a combination of both .groupby() and .describe() like this

df.groupby('Category')['Score'].describe()

and this almost looks like what I want but when I come to view this as a Dataset, all of the stats are in the index. I would like the data to be in the form of a table so I can output it and create a visualization off of the back of it.

Any ideas?

Thanks


texts: How to score arbitrary sentences based on how likely they are to occur in the real world?

texts:
According to article about LSTM here, I know that:

  it allows us to score arbitrary sentences based on how likely they are to occur in the real world. This gives us a measure of grammatical and semantic correctness. Such models are typically used as part of Machine Translation systems.

But, it seems that this article doesn't point out how to compute the score with LSTM.

Any way to compute the score?


texts: Which NN architecture solves my problem?

texts:
I want to convert images frame into text, for instance, if I have a image with a dog that plays with a ball I will produce a simple text "Dog plays with ball" (Ok, I know I must have x samples training, but I mean ipotetically).

Which type of NN allows me to perform that action?

Thank you


texts: Does gradient descent always converge to an optimum?

texts:
I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.

I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am also aware that it might diverge from an optimum if, say, the step size is too big. However, it seems to me that, if it diverges from some optimum, then it will eventually go to another optimum.

Hence, gradient descent would be guaranteed to converge to a local or global optimum. Is that right? If not, could you please provide a rough counterexample?


texts: Reason for adding 1 to word index for sequence modeling

texts:
I notice in many of the tutorials 1 is added to the word_index. For example considering a sample code snippet inspired from Tensorflow's tutorial for NMT https://www.tensorflow.org/tutorials/text/nmt_with_attention :
import tensorflow as tf
sample_input = ["sample sentence 1", "sample sentence 2"]
lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
lang_tokenizer.fit_on_texts(sample_input)
vocab_inp_size = len(lang_tokenizer.word_index)+1

I dont understand the reason for adding 1 to the word_index dictionary. Wont adding a random 1 affect the prediction. Any suggestions will be helpful


texts: Which outlier detection can detect these outliers?

texts:
I have a vector and want to detect outliers in it.

The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal.

I need an outlier detection method (a non-parametric method) which can just detect red points as outliers. I tested some methods like IQR, standard deviation but they detect yellow points as outliers too.

I know it is hard to detect just the red point but I think there should be a way (even combination of methods) to solve this problem.

Points are readings of a sensor for a day. But the values of the sensor change because of system reconfiguration (the environment is not static). The times of the reconfigurations are unknown. Blue points are for the period before reconfiguration. Yellow points are for after the reconfiguration which causes deviation in the distribution of the readings (but are normal). Red points are results of illegal modifying of the yellow points. In other words, they are anomalies which should be detected.

I'm wondering whether the Kernel smoothing function estimate ('pdf','survivor','cdf',etc.) could help or not. Would anyone help about their main functionality (or other smoothing methods) and justification to use in a context for solving a problem?


texts: Most Efficient Post Processing with Python and Pandas

texts:
This question is about best practices for working in Pandas dataframes.  Speed, ease of use, and memory consumed could all impact any answers you might have.  I start by pulling a data set into a dataframe like this:

Date    Location    Value
3/4/2018    1        4795
3/5/2018    1        4795
3/4/2018    2        5022
3/5/2018    2        5088
3/4/2018    3        100
3/5/2018    3        100
3/4/2018    4        117154
3/5/2018    4        117154

I would like to sum the Value based on some other criteria.  For this example, lets use two states, SD and ND.  Location 1,2,4 are in ND, and Location 3 is in SD.  As I see it, I have two options:

Have Pandas post process the location numbers.  IE Pythonicly: ND = Sum(Loc 1,2,4), SD = Sum(Loc 4).  Then build/pivot the time series based on state
Build a lookup table and have Pandas append the state to each row in the dataframe.  Then filter/group by state for the time series.  Lookup table would look like so:

Location    State
1           ND
1           ND
2           ND
2           ND
3           SD
3           SD
4           ND
4           ND

In option one, would Pandas add a new row to the dataframe for each day of the timeseries with the state totals?  Or would the output be a pivot table like structure with only the state totals.

If option two, what would be the best type of way to host the lookup table?  CSV?  JSON?  Table in SQL DB?

I'm concerned in option one changes would need to be made directly to the code.  Whereas in option two an addition to the lookup table could add the information required to add location to the correct group.

While I know this is open ended, I hope someone is willing to provide thoughts on efficient structure for this type of data flow.


texts:
I build model that on input have correct word. On output there is possible word written by human (it contain some errors). My training dataset looks that:

input - output
hello - helo
hello - heelo
hello - hellou
between - betwen
between - beetween
between - beetwen
between - bettwen
between - bitween

etc.
During preprocessing I add a measure of the distortion of a word. Then I hardcoding letters for numbers.
My current model's using CNN. The number of neurons of input is the same as the longest word in training dataset and the number of neurons of output is the same as the longest word in traning dataset.
This model doesn't work as I excepted. Word on the output is not look as I except.
eg.

input - output
house - gjrtdd

Question:

How can I build/improve model for this task? Is CNN a good idea? What other methods can I use for this task?


texts: Binary classification to predict various targets

texts:
I am using keras neural networks for a binary classification task. I have a large dataset with 19 columns. Each of these columns is of binary type, and therefore the entire dataset is just 1s and 0s, like so (obviously this is not the entire dataset, which has 19 columns and many more rows):

I want to make 19 predictions, one for each column, where the other 18 are used as independent variables. So essentially, first I'll make col 1 the target, using the other 18 columns to predict it. Then I'll make col 2 the target and use the other 18 to predict it etc etc...until column 19. Is there any way to do this effectively without making 19 separate models for each column? Thanks!


texts: Run queries against results of previously cached queries

texts:
I'm trying to find a tool or workflow that would allow a team to analysis data in multiple steps, where intermediate results are cached. (I'll assume that the raw data is in a relational database and the end result shall be shown in a web UI.)

Multiple steps: This requirement can be describe in SQL as VIEWS:

CREATE VIEW b AS SELECT FROM a WHERE ... GROUP BY ... HAVING ...
CREATE VIEW c AS SELECT FROM b WHERE ... GROUP BY ... HAVING ...

Caching: Additionally to the process described above, we want to temporarily store the 'views' b and c. (Either until the underlying data changes or for a set amount of time. The latter is acceptable for the use case.)

Saving the view as a (temporary) tables, updated periodically, comes to mind, but we would have to build a workflow around this.

Team: This project requires some collaboration. A user should be able share queries with a team and have others build on top of it.

I want to avoid reinventing the wheel. Is there a technology stack that does this already?


texts: Splitting and training multiple datasets at the same time

texts:
I've got 15 different datasets at about 10GB each. Each dataset comes with a binary 2D ground truth (10486147ish, 1) that I pull from it. I'm trying to figure out how to load each dataset, split them all with scikitlearn's train_test_split, then iterate over all 15 datasets per epoch. Under normal circumstances, the datasets would be shuffled as well, but I cannot figure out how to even do that since the data is too large to load all at once to shuffle (as such shuffling them is on the back burner for now).

Here's what my code looks like for one dataset.

import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from sklearn.preprocessing import sequence
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection imporrt train_test_split

arr = np.load ('source/dir/dataset1.npy', allow_pickle = True, fix_imports = True)
arr[arr == -inf = -9999]
rehape = arr.reshape(((arr.shape[0])*(arr.shape[1])), (arr.shape[2]))
drop = reshape[~np.all(reshape == -9999, axis = 1)]
#additional work done with -9999 here
truth = drop[:,46]
data = drop[:,0:45]

#callbacks deleted in code sample

encoder = LabelEncoder()
encoder.fit(truth)
Y = encoder.transform(truth)
Y = Y.reshape(10486147, 1)
X = data.reshape(10486147, 45, 3)

seed = 7
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.33, random_state = seed)

model = sequential()
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3)))
model.add(Dense(1, kernel_initializer = 'normal', activation = 'sigmoid'))

model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 500, batch_size = 1000, callbacks = [deleted callbacks])

So that makes sense for one dataset, but as I've said before I've got 15 datasets to iterate through, and I don't think retraining on new data is the right step.
Is there a way through dataset1.npy through dataset15.npy while properly splitting the ground truth as well.

Any suggestions?


texts: Relationship between KS, AUROC, and Gini

texts: Common model validation statistics like the Kolmogorov–Smirnov test (KS), AUROC, and Gini coefficient are all functionally related. However, my question has to do with proving how these are all related. I am curious if anyone can help me prove these relationships. I haven't been able to find anything online, but I am just genuinely interested how the proofs work. For example, I know Gini=2AUROC-1, but my best proof involves pointing at a graph. I am interested in formal proofs. Any help would be greatly appreciated!


texts: Why do we need data scientists if there are websites like this?

texts:
Very basic question about data science as I'm a beginner. I noticed that there are sites like illustrated below, that help visualize and analyze data so that the business owner can have a better understanding of their business.
My question is that if there are sites like this, why are data scientists needed? What can they offer that sites like this can't?


texts: How would you quantify an experience into a score without labeled data

texts:
How would you approach a scenario where you have to quantify an abstract notion like “customer experience” without having any labeled data? So basically what you have are bunch of variables that you know, more or less, how they affect the experience, but you don't know the "weights" of importance of each one.

So e.g. if it is an experience from a food delivery service, then I have the ETAs for the order, ratings (not very reliable by the day), cancelation reasons, $ amounts, etc.


texts: Understanding output probabilites of xgboost in multiclass problems

texts:
I would like to understand the output probabilities of a xgboost classifier (or any other decision tree ensemble based classifier) in the case of a multiclass problem.
For example: We have 5 different classes and a trained model on some data belonging to those classes. I would expect, given some test data, the model would give output probabilities of up to 100% when it is really sure the data belongs to some class. Obviously the probabilites still have to add up to 1.
On the contrary I was told that in the case of 5 different classes, the model would be already sure when outputting 20% (100/5) for the data. I also see this in a problem I am facing, the probabilities are never higher than 30%.
I would like to understand this apparent fact, ideally pointing me to a source where this is explained.


texts: Spread of infection: how to correct for population size

texts:
The Covid-19 death data for different countries should reflect how fast the infection spreads through the population and how susceptible the population is.
The total number of deaths after some longer time will depend on the population size but the initial infection rate less so.

For example if the infection starts with one person, then the speed at which it spreads will not depend on population size per se (it will depend on density and connected-ness).

However, in a larger population it is more likely there will be more then one patient zero.
Also after some time the number of new infections slows down because there are fewer not-infected individuals left in a network. In a larger population this mitigating effect will be smaller because there are more networks to infect.

Taking the above into consideration, how should the numbers on covid-19 deaths be corrected for population size?


texts: How to plot an image from a series of bits inPython?


texts: Orange Biolab Pivot Table widget - missing

texts:
I downloaded most recent version of Orange. I can not find the Pivot Table widget anywhere. When right clicking on the search menu shown, it does not appear anything when writing Pivot Table phrase.

Here is the link to the documentation on this widget:

https://orange3.readthedocs.io/projects/orange-visual-programming/widgets/data/pivot.html


texts: Anomaly detection on text data using one Class SVM

texts:
I am working on an Anomaly detection problem that is first of its kind for me. I am able to do data cleansing and train the model however I have no idea on predicting results, as it is new for me and I am running out of time.

My notebook and data is available on Kaggle here:
https://www.kaggle.com/sanjeet7/webgateway

Any help will be deeply appreciated.


texts:
I've trained 4 classifiers on an undersampled dataset.
I plotted the learning curve for each classifier and I got the following results :

I see that for the Log Reg, both curves seem to converge and that adding more data will not help at some point.
For the SVC I have no idea (rather than adding more data seems good ! )
for Knn : adding more data will increase both accuracy
for Random Forest : I have no idea.
I would love to understand how to read these curves. Thank you very much ! :)


texts: Optimal ML method to detect coupling in tabulated data

texts:
Consider a dataset with these features

state (s)
store_id (s_id)
transaction_id (t_id)
product_id (p_id)
unit_price (p_p)

(s),(s_id),(t_id),(p_id),(p_p)
"MN", 111, 222, 987, 1.23
"MN", 111, 222, 987, 1.23
"MN", 111, 222, 222, 2.99
"MN", 123, 333, 987, 1.25
"MN", 123, 333, 222, 2.99
"NV", 567, 345, 987, 0.23
"NV", 567, 345, 987, 0.23
"NV", 567, 345, 987, 0.23
"FL", 345, 333, 222, 9.99
"FL", 345, 333, 222, 9.99

(We have some billion rows like this)

We can see that MN tends to buy 987 together with 222, whereas other states buy those separately. We also see that MN customers are very sensitive to price: people buy two items at a price of \$1.23, but only one item at a price of \$1.25.
Both NV and FL tend to buy 987 and 222 separately.

My question is. Which tool is optimal for this analysis? Decision trees? XGBoost? NN?


texts: Recurrent Neural Network (LSTM) not converging during optimization

texts:
I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one hot encoded before being used and I am using the Adam optimizer which is implemented like this.

   for k in M.keys(): ##For k in weights
        M[k] = beta1 * M[k] + (1-beta1)*grad[k]
        R[k] = beta2 *R[k] + (1-beta2)*grad[k]**2
        m_k = M[k] / (1-beta1**n)
        r_k = R[k] / (1-beta2**n)
        model[k] = model[k] - alpha * m_k / np.sqrt(r_k + 1e-8)

Beta1 is set to 0.9, beta2 to 0.999 and alpha is set to 0.001. When I train it for 50,000 I get very high fluctuation of the cost and it never seems to significantly decrease (only sometimes due to the fluctuations (and I catch the weights with the lowest cost)).
After sketching the cost of iterations I get a graph like this:

It seems to be increasing on average only seeming to decrease to the the large fluctuations. What can I change to have better success and have it converge?

Thanks for any help


texts: Are 3D kernels in convolutions summed over their channels?

texts:
Say for example that I have a 28x28x1 grey scale image and I will perform two consecutive convolutions. The first convolution has 2 3x3x1 filters and the second has 3 3x3x2 filters. Each convolution is done without padding and with a stride of 1. According to what i have read online the output after both convolutions would in this case be 24x24x3, if i've understood it correctly. The only way i understand that we could get this shape is by summing the 3D kernels over their channels, like we do when convolving over a RGB image. I've demonstrated what i mean in the image below:

It is the step were we go from 3 24x24x2 images to 3 24x24x1 images that i'm wondering about. Is this how convolutions are performed, by summing over the channels (2 channels in this case)? If not, how do we get the output shape of 24x24x3?


texts: How to implement Python's MLPClassifier with gridsearchCV?

texts:
I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code:

parameters={
'learning_rate': ["constant", "invscaling", "adaptive"],
'hidden_layer_sizes': [(100,1), (100,2), (100,3)],
'alpha': [10.0 ** -np.arange(1, 7)],
'activation': ["logistic", "relu", "Tanh"]
}

clf
= gridSearchCV(estimator=MLPClassifier,param_grid=parameters,n_jobs=-1,verbose=2,cv=10)

Though,I am not sure if hidden_layer_sizes: [(100,1), (100,2), (100,3)] is correct. Here, I am trying to tune 'hidden layer size' & 'number of neurons'. I would like to give this 'tuple' parameter for hidden_layer_sizez: 1, 2, 3, and neurons: 10, 20, 30,...,100.

But
I do not know if it is the correct way to do it. Therefore, I am choosing default neurons to be 100 in each layer.

Can
anyone advise please?


texts: How to treat outliers in a time series dataset?

texts:
I've read the following article about how to treat outliers in a dataset: http://napitupulu-jon.appspot.com/posts/outliers-ud120.html

Basically, he removes all the y which has a huge difference with the majority:

def outlierCleaner(predictions, ages, net_worths):
    """
        clean away the 10% of points that have the largest
        residual errors (different between the prediction
        and the actual net worth)

        return a list of tuples named cleaned_data where
        each tuple is of the form (age, net_worth, error)
    """

    #calculate the error,make it descend sort, and fetch 90% of the data

    errors = (net_worths-predictions)**2
    cleaned_data =zip(ages,net_worths,errors)
    cleaned_data = sorted(cleaned_data,key=lambda x:x[2][0], reverse=True)
    limit = int(len(net_worths)*0.1)

    return cleaned_data[limit:]

But how may I apply this technique to a time series dataset if its rows are correlative?


texts: Aggregate categorical feature by the target

texts:
Having a list of triplets {X1,X2,Y} such as :

{pennsylvania, fever , malaria}
{pennsylvania, headache  , malaria}
{arizona, ketone smell  , flu}
{new york, fever  , cancer}
{ohio, hand pain  , trauma}

i have thousands of samples with states , symptoms and diagnoses.
i need to aggregate the states to reduce the dimension into smaller set of states using the symptoms and the diagnose. Any idea for that?

i started using Kmode algorithm for clustering the data into several clusters but i'm not sure if it makes any sense doing that


texts:
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...).
I am using  a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .

def FPR(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    result = fp / (fp+tn)
    return result
def FNR(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    result = fn / (tp+fn)
    return result
FPR_scorer = make_scorer(FPR)
FNR_scorer = make_scorer(FNR)

def get_CrossValResults2(model,cv_rst,bestIndx):
    best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]
    roc="{:.12f}".format(best['mean_test_roc_auc'])
    acc ="{:.0%}".format(best['mean_test_accuracy'])
    prec ="{:.0%}".format(best['mean_test_precision'])
    rec ="{:.0%}".format( best['mean_test_recall'])
    f1 ="{:.0%}".format(best['mean_test_f1'])
    r2="{:.2f}".format(best['mean_test_r2'])
    g_mean="{:.2f}".format(best['mean_test_gmean'])
    pr_auc="{:.8f}".format(best['mean_test_pr'])
    fnr="{:.0%}".format(best['mean_test_fnr'])
    fpr="{:.0%}".format(best['mean_test_fpr'])
    rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])
    return rst

cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])

scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}
param_grid = {'n_estimators': [200],
             'max_depth': [80,90],
              'min_samples_leaf': [2,3, 4],
            'min_samples_split': [2,5,12],
            'criterion': [ 'gini'],
             'class_weight' : [class_weights], 'n_jobs' : [-1]}
clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model
bestmodel = clf.fit(X,Y)

cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)


texts: Why keep vocabulary and posting list separate in a search engine

texts:
I am taking a class in information retrieval. We learned that the index of a search engine has (possibly among other things):

A vocabulary mapping terms to their statistics (frequency, type, ...) and
A posting list mapping terms to the documents were they are stored (with or without positions, fields, ...)

These are separate data structures. I understand why those information is needed and what for. But I don't understand why we want to keep them separate. Why can't we have one data structure that maps terms to statistics and documents?

I am currently thinking it might be because the vocabulary would be much smaller and we could read it from memory. So we could use the statistics to remove certain query terms, which are likely not useful or to try to find misspellings in the query without having to touch the large posting list.

Is this correct or is there another reason to keep vocabulary and posting list separate?


texts: TFIDF and TFIDF weighted W2V with Multinomial Naive Bayes?

texts: Although the tfidf vectors don't really follow Multinomial Distribution, yet MultinomialNB works fairly well, why is it so? Also would weighted tfidf w2v work the same way or should I use GaussianNB for that? plus since w2v vectors can be negative too, so it makes sense to use minmax scaler on it. but this would then affect be affected in presence of outliers. Also i had read somewhere that doing processing on text vectors can be actually bad because they may not represent the actual text then.


texts: How to handle large number of categories in a dataset?

texts:
I have one dataset of "Books" which contains 8 columns initially and out of which 3 of them contains text values which can be categorized. The 3 columns contains "Language-code", "Author Name" and "title" of the book. As sklearn LinearRegression don't take text as input so i decided to categorize these 3 columns by using "pandas_getdummies(...)" but after categorizing it the columns number exceeded to 20072 from 8 which is way too high.

The dataset url is: https://www.kaggle.com/jealousleopard/goodreadsbooks/downloads/goodreadsbooks.zip/6

So my queries are:

What to do with the title name? Categorizing it doesn't seems right.
What to with the rest 2 columns? If i leave the title name then the number of columns exceeds to 7646. Is there any other algorithm where i can directly feed the dataset without categorization?
How to handle these large number of features after categorizing?


texts: Tools and protocol for reproducible data science using Python

texts:
I am working on a data science project using Python.
The project has several stages.
Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and creating another data set.
I store the code in git, so that part is covered.
I would like to hear about:

Tools for data version control.
Tools enabling to reproduce stages and experiments.
Protocol and suggested directory structure for such a project.
Automated build/run tools.


texts: K-Means visualisation problem 8 numerical feutres

texts:
Hello im looking for exemple with python for K-Means clustering when i have data set with more than 6 feutres.
thanks


texts: How to draw Deep learning network architecture diagrams?

texts: I have built my model. Now I want to draw the network architecture diagram for my research paper. Example is shown below:


texts:
I am dealing with acoustic data with very high sampling frequency of 2MHz and want to build a classifier.

I was wondering if there are any rules of thumb for preprocessing acoustic data. Is it better to directly use raw data (timesignal) or first to construct spectrograms, and to use these?

There are papers, which say raw is better, and there are papers saying spectrograms are better. It somehow seems to me, that authors already had a preferred method, even before writing the paper. I think a real comparison is difficult.

I read the paper "Deep Learning and Its Applications to Machine Health Monitoring: A survey", in which a study of different methods was done. I looked up his references, but authors seemed just to pick raw or spectrograms without explaining. For example, in the paper "End-to-end learning for music audio" from Dieleman spectrograms are preferred. In "Sample-Level deep convolutional neural networks for music auto-tagging using raw waveforms", they claim their 1D structure is better or at least comparable to 2D architectures.

Personally I had better experience with spectrograms.


texts: What is the concept of Normalized Mutual Information in the evaluation of Clustering?

texts:
I know what mutual information basically is but not quite sure about why and how it is used in the context of evaluation of clustering mechanisms ? Can someone please explain the intuition behind it ?
i.e, how it is defined in the case of clustering evaluation ?


texts:
I was reading about descending gradient.
How does the descending gradient know what weights to adjust?
Does it adjust to all network weights at the same time?
Does each weight have an associated error?


texts: Algorithm for deriving multiple clusters

texts:
Suppose I have a set of data (with  2-dimensional feature space), and I want to obtain clusters from them. But I do not know how many clusters will be formed. Yet, I want separate clusters (The number of clusters is more than 2). I figured that k means of k medoid cannot be used in this case. Nor can I use hierarchical clustering. Also since there is no training set hence cannot use KNN classifier to any others (supervised learning cannot be used as no training set). I cannot use OPTICS algorithm as I do not want to specify the radius (I don't know the radius) Is there any machine learning technique that would give me multiple clusters (distance based clustering) that deals well with outlier points too?
This should be the output:


texts: Ideal strategy for multi variable regression attempting to maximize the target

texts:
I am trying to implement machine learning for the following data

Data Input

What I am trying to achieve is to keep the ad bid & cost per sale as low as possible while increasing sales.

This is a multi variable, multi response regression and I am looking for ideas and machine learning models to implement this.

I don't think reinforcement or multi-output regression will work because I want to reduce ad bid + cost per sale and at the same time increase sales (not target)

Technology/SDK is not a problem. I am looking for architecture/model ideas.


texts: Categorise sentences based on their semantic similarity

texts:
I have a set of unique sentences. For each sentence I calculate a semantic similarity score (between 0 to 1) with the remaining sentences as mentioned in the below example.

E.g.,

Dataset = {sen1, sen2, sen3, sen4,..., senN}

For sen1 I calculate pairwise semantic similarity scores as follows.

sen1 and sen2 = 0.3
sen1 and sen3 = 0.7
sen1 and sen4 = 0.9
...
...
...
sen1 and senN = 1.0

Likewise for all the sentences I calculate pairwise semantic similarity scores.

Since, I am getting a pairwise value, is it possible to cluster these sentences? Also what is the most appropriate clustering technique in my situation?

(I want to cluster sentences based on the similarity value I have and also I consider values above 0.5 as semantically similar sentences.)


texts: Image segmentation for small images with rectangles

texts:
I am doing toy project and trying to make image segmentation for rectangles generated in small images. Here the code:

import numpy as np
from PIL import Image, ImageDraw
import random
import cv2

GRID_SIZE = 32

def generate_image_and_mask(self):
    random.seed(100)
    #newIm = np.zeros((GRID_SIZE, GRID_SIZE, 3), np.uint8)
    img = Image.new('RGB', (GRID_SIZE, GRID_SIZE), color='black')
    draw = ImageDraw.Draw(img)

    randX = random.randint(0, GRID_SIZE)
    randY = random.randint(0, GRID_SIZE)

    w = random.randint(0, GRID_SIZE)
    w = min(w, randX - GRID_SIZE)

    h = random.randint(0, GRID_SIZE)
    h = min(h, randY - GRID_SIZE)
    draw.rectangle([randX, randY, w, h,], fill='white')
    destBGR = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
    return destBGR, np.array(img.convert('L'))

Because of upsampling operation in the decoder rectangles corners look rounded( trained deeplabV3 model) :

I can increase picture side from 32 to 1200 and it will looks good, but that is a bad solution. What do I need to do to get better rectangle mask for small images? How to change the network architecture? The same story is with other segmentation models, not only deep lab.


texts: How to interpret two continous variables output using GAM?

texts:
I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (count variable). I have tried a lot but I am not able to understand how to interpret the graph and output that I am getting.
I tried this formula using mgcv package:
model1<- gam(cvd ~ s(templg0), family=poisson)
summary(model1)
plot(model1)

So here is the output for summary that I am getting:
Family: poisson
Link function: log

Formula:
cvd ~ s(templg0)

Parametric coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept) 3.195669   0.004877   655.2   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Approximate significance of smooth terms:
             edf Ref.df Chi.sq  p-value
s(templg0) 3.422  4.295  57.23 2.93e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

R-sq.(adj) =  0.0152   Deviance explained = 1.68%
UBRE =  1.016  Scale est. = 1         n = 1722

Can someone please explain the output in detail. What this output is explaining? and also can someone help what this plot (picture attached) is showing? Please be kind as I have invested a lot of time but can not find how to interpret this.


texts: What's a good F1-score in (not) extremely imbalanced dataset?

texts:
I have a dataset with around 4.7K focused on binary classification. Class proportion is 33:67. meaning Label 1 is 1558 (33%) and Label 0 is 3154 (67%) of my dataset.

Is my dataset imbalanced? some people say it is not bad

My objective is to increase the F1-score only. I set the class_weight=balanced in my parameters and scoring=f1 during CV as shown below.

svm=SVC(random_state=42)
svm_cv=GridSearchCV(svm,param_grid,cv=5,scoring='f1')
svm_cv.fit(X_train_std,y_train)

Can you let me know through a code sample on how I can increase the weightage to minority class? if that is any different from choosing balanced paramter

Currently my results are like as follows

I understand AUC for few algo is above 80 but I believe F1-score is more important for a imbalanced class problems like mine.

Can you help? I tried Oversampling minority class but not much improvement.

Increasing features doesn't take me to 80% F1-score


texts: Reward negative derivative on linear regression

texts:
I'm actually new to Data Science and I'm trying to make a simple linear regression with only one feature X ( which I added the feature log(X) before adding a polynomial features) on a motley dataset using Python an all the Data Science stack that comes with it (numpy, pandas, sci-kit learn, ...)

Here you can find a piece of code of my regression using scikitlearn:

def add_log(x):
    return np.concatenate((x, np.log(x)), axis=1)

 # Fetch the training set
_X = np.array(X).reshape(-1, 1) # X = [1, 26, 45, ..., 100, ..., 8000 ]
_Y = np.array(Y).reshape(-1, 1) # Y = [1206.78, 412.4, 20.8, ..., 1.34, ..., 0.034]
Y_train = _Y
X_train = add_log(_X) if use_log else _X

# Create the pipeline
steps = [
    ('scalar', StandardScaler()),
    ('poly', PolynomialFeatures(6)),
    ('model', Lasso(alpha=alpha, fit_intercept=True))
]

pipeline = Pipeline(steps)
pipeline.fit(X_train, Y_train)

My feature X can go between 1 to ~80 000 and Y can go between 0 and ~2M

There is one thing I know about the curve I should obtain is that it should always decrease so the derivative should be always negative

I make a little schema to explain what I expect vs what I have:

Therefore I would like to reward prediction where derivative is always negative even if my data suggest the opposite.

Is there a way to do that with sci-kit learn?
Or maybe I'm suggesting a bad solution to my problem and there is another way to obtain what I want ?

Thank you


texts: Benchmark datasets for collaborative filtering

texts:
I'd like to test a new algorithm for collaborative filtering. A typical use case is to recommend movies based on the preferences of users similar to the specific user.

What are some common benchmark datasets that researchers often use to test their algorithms? I know that within Computer Vision people often use MNIST or CIFAR, but I haven't found similar datasets for collaborative filtering.


texts: How to do k-folds in python whilst splitting into 3 sets?

texts:
Consider the following data:

   import pandas as pd
    wine = pd.read_csv(r'wine_data.csv', names = ["Cultivator", "Alchol", "Malic_Acid", "Ash", "Alcalinity_of_Ash", "Magnesium", "Total_phenols", "Falvanoids", "Nonflavanoid_phenols", "Proanthocyanins", "Color_intensity", "Hue", "OD280", "Proline"])
    X = wine.drop('Cultivator',axis=1) #input
    y = wine['Cultivator'] #output

y is what i am trying to predict and X is the input and i will be using some sort of mlp classifier. What I want to do is split this data into test, training and validation and then apply K-folds. I'm struggling to see how you do this..

I know that I can obtain validation, test and training by using the following:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=1)

But what I want to do now is apply k folds such that for each fold I have 3 sets: validation, testing , training rather than just 2 sets.

I know I can use the below for Kfolds:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)
X_np=np.array(X)
y_np=np.array(y)

After converting to a numpy array I can then do this:

for train_index, test_index in kf.split(X_np):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X_np[train_index], X_np[test_index]
    y_train, y_test = y_np[train_index], y_np[test_index]

But how do I get a 'validation_index'. In general the question is how do I use k-folds when I have 3 sets as opposed to just 2?

Also when do I normalize the data; do I normalize when i've split into X_train, X_test as above... or do I do it before?

Any help appreciated.


texts: Detecting off state in the magnitude of accelerometer data?

texts:
I have a univariate time series signal. It's the magnitude of an accelerometer attached to an engine.

I need to create an algorithm to detect off state, please see the black lines in the image below. The rest of signal is comprised of idle and active states. Idle state looks slightly higher and active state tend to have huge spikes and generally higher mean than both idle and off.

While this is simple, several machines have different mean values based on the size of the engine and the proximity of the sensor to the vibration. So the idea of using thresholds will not help.

I have considered K-Means algorithm. It worked pretty well when three states are available. When one or two are absent, the results are degraded significantly since it attempts to find two classes that don't exist in the data.

I have tried Hidden Markov Models. They looked promising, unfortunately they train themselves to identify the distribution of the states which will again change from one machine to another.

I thought of using standardisation. But I'm hesitant since the mean value of off state will change accordingly.

What unsupervised or semi-unsupervised approaches do you recommend on detecting off/idle/on states?


texts:
I want to create a drug using N different chemicals for fighting a bacterial infection those N chemicals are contained inside the drug in different quantities my work environment is a simulated one and I want to create the drug using genetic algorithms.
How can I use genetic algorithm for this problem. What's representation for chromosomes can I use and what kind evaluation function should I use?
What could go wrong using genetic algorithms for these kinds of problems?


texts: I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid noticing a few things. With my math background, I know that if I'm trying to find the global min/max of a function, I must first find all the critical points first. The course talks about convergence of GD, but is it really guaranteed to converge to the global min? How do I know it won't get stuck at a saddle point? Wouldn't be safer to do a 2nd derivative test to test it? If my function is differentiable it seems reasonable it converges to a local min, but not to the global min. I have tried looking for a better explanation but everyone seems to take it for granted without questioning. Can someone point me in the right direction?


texts: Sci-kit learn function to select threshold for higher recall than precision

texts:
When we care more that there should be no false negatives, as far as possible… ie. higher recall (video is suitable for kid or not), we should use (receiver operating characteristic) ROC (area under the curve) AUC and try to maximize it.

Scikit-Learn provides a function to compute this directly:

from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)

Similarly, when we care more about the false positives than the false negatives, example shop lifting case then what should we do?

Should we just try to maximize the recall ignoring the precision or is there a better metric?

I had read somewhere "Use the precision vs recall curve and get that max" but not very sure what that means... Can you please explain the same?

If there a direct function for this metric in sci-kit learn, like above for the first case, please do let me know.


texts:
-1
down vote
favorite
I have a dataset including missing data for most of the variables. Assume the dataset is as follows:

Obs.  var1  var2  var3  var4  var5  var6
1     x11   x12   x13   x14    Nan  Nan
2     x21   x22   x23   Nan    x25  x26
3     x31   x32   x33   x34    x35  x36
...
n     xn1   xn2   xn3   xn4    Nan  xn6

I have split the dataset to d1 where we have complete data for all variables and d2 where all records have at least one missing variable.

I made different models using KNN: To predict the values of var5 and var6 for the first observation, I used d1 (dataset without missing value) and modeled on var1, var2, var3 and var4.

To predict the value of var5 for the last observation, I used d1 and modeled on var1, var2, var3, var4, and var5.

Does my approach make sense?! Any suggestion are welcome. Thank you.


texts: What is the difference in plotting ROC curve with probability scores vs binary decisions

texts:
As the title reads what is the difference? Plotting the ROC w.r.t probability scores gives the stair cased version.
But in my opinion I find that using binary decision is better because the ROC curve is just a summary of the different confusion matrices w.r.t different thresholds. And the confusion matrix shows the binary decisions.
Is their an advantage or certain situation where plotting the ROC curve with the prob.scores is better than the binary decisions?


texts: rows to columns in data.table R (or Python)

texts:
This is something I can't achieve with the reshape2 library for R. I have the following data:

 zone       code        literal
 1: A         14           bicl
 2: B         14           bicl
 3: B         24          calso
 4: A         51           mara
 5: B         51           mara
 6: A         125           gan
 7: A         143          carc
 8: B         143          carc

i.e.: each zone has 4 codes with its corresponding literal. I would like to transform it to a dataset with one column for each of the four codes and one column for each of the four literals:

 zone  code1 literal1   code2 literal2   code3 literal3   code4 literal4
 1: A    14     bicl      51     mara     125      gan      143   carc
 2: B    14     bicl      24    calso      51     mara      143   carc

Any easy way to achieve this in R? If not, I would also be comfortable with a solution in Python.


texts: Any usable libs to build and visualise SOM in python?

texts:
I tried SOMpy, though it is very crude now and works only with oldest versions of matplotlib.
Is there any fancy lib that can build SOM based on array and visualize it in Python?


texts: Why does Faster R-CNN use SGD optimizer instead of Adam?

texts:
I just start learning Faster R-CNN and I have some doubts about the optimizer of this network. In my understanding, Adam optimizer performs much better than SGD in a lot of networks. However, the paper of Faster R-CNN choose SGD optimizer instead of Adam and a lot of implementations of Faster R-CNN I found on github use SGD as optimizer as well.
I guess that in case for Faster R-CNN Adam maybe doesn't have a better performance. After I looked up for my guessing, I found this answer link that gave me a rough idea. In the answer, it suggests that SGD is a better generalized adapter than ADAM. But I still need some more detailed explannations about it.
Here is my questions:

Can we use Adam as optimizer for Faster R-CNN? If someone has used Adam for Faster R-CNN, could you share some results about Adam's performance?
As the answer in the link above suggests, Adam may have worst performance in some special cases. I would like to ask in what kinds of special cases will Adam perform poorly. Can anyone gives me some examples? And does Faster R-CNN belongs to these special cases?


texts: What is local-NMF? How is it better than original NMF?

texts:
I am reading this paper, but don't really understand.

Do the words "part-based" or "local" for non-negative matrix factorization (NMF) mean that the algorithm aims to factorize some specific parts more than other unimportant parts?

Like the algorithm tries to seek some max signals in matrix and focus on that part? Is there a simple explanation about local-NMF?


texts:
I am trying to train a LSTM model. Is this model suffering from overfitting?

Here is train and validation loss graph:


texts:
I trained the data this way :
There are four classes, the data distributed evenly (same amount of labels).

Used min_max_scaler
Used train_test_split(X,y,test_size=0.3,random_state=42,stratify=y)
Ran GradientBoostingClassifier - once with n_estimators=32, and once with n_estimators=500 on the training data
Used predict on the test data
Got accuracy=0.94 on n_estimators=32 and accuracy=1 on n_estimators=500.
Precision and recall from classification report is also 1 for all class

Seems fishy but I can't figure out why... what am I doing wrong ?


texts:
I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to generate the training data by convolve  the image of license plat letter with kernels (somethings like gaussian blur,box blur) using python, but it does not resemble real data, here is the font look like:

and I want to generate individual letter, that look like this:

Do you have any idea how to generate training data like above picture, ?
thank you


texts: How do I Combine two CNN models (h5 format)?

texts:
I have two CNN models, both of them are trained on the same dataset.

How do I combine/ensemble both to make predictions on test data?

# Load Keras Models

model1 = tf.keras.models.load_model('/kaggle/input/models/model1.h5')
model2 = tf.keras.models.load_model('/kaggle/input/models/basic_cnn.h5')

How to combine these 2 models to make model3 ?

I have to use predict_generator function to predict from the ensembled model3.


texts: Data science without knowledge of a specific topic, is it worth pursuing as a career?

texts:
I had a conversation with someone recently and mentioned my interest in data analysis and who I intended to learn the necessary skills and tools. They suggested to me that while it is great to learn the tools and build the skills there is little point in doing so unless i have specialized  knowledge in a specific field.

They basically summed it to that I'd just be like a builder with a pile of tools who could build a few wooden boxes and may be build better things (cabins, cupboards etc), but without knowledge in a specific field I'd never be a builder people would come to for a specific product.

Has anyone found this or have any input on what to make of this ? It would seem if it was true one would have to learn the data science aspects of things and then learn  a new field just to  become specialized.


texts: Bootstrapped curves and machine learning

texts:
I am exploring machine learning models to employ for my use case as described below and would appreciate some expertise

Effectively I have bootstrapped bond prices to derive a zero coupon curve for the obligor however when comparing it with the Bloomberg curve for that same obligor my curve contains lots of volatility and outliers in comparison

I seek to use a ML model which would
Fit the bootstrapped curve to resemble that of Bloomberg.

Any ideas on what model can be used here to adapt my bootstrapped curve to the Bloomberg one?


texts: Should input images be normalized to -1 to 1 or 0 to 1

texts:
Many ML tutorials are normalizing input images to value of -1 to 1 before feeding them to ML model. The ML model is most likely a few conv 2d layers followed by a fully connected layers. Assuming activation function is ReLu.

My question is, would normalizing images to [-1, 1] range be unfair to input pixels in negative range since through ReLu, output would be 0. Would normalizing images into [0, 1] range instead be a better idea?

Thank you.


texts: Sklearn ValueError: X has 2 features per sample; expecting 11

texts:
I try to visualizing multiple logistic regression but I get the above error.

I'm practicing on red wine quality data set from kaggle.

Here is a full traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-88-230199fd3a97> in <module>
      4 X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
      5                      np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
----> 6 plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
      7              alpha = 0.75, cmap = ListedColormap(('red', 'green')))
      8 plt.xlim(X1.min(), X1.max())

/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/base.py in predict(self, X)
    287             Predicted class label per sample.
    288         """
--> 289         scores = self.decision_function(X)
    290         if len(scores.shape) == 1:
    291             indices = (scores > 0).astype(np.int)

/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/base.py in decision_function(self, X)
    268         if X.shape[1] != n_features:
    269             raise ValueError("X has %d features per sample; expecting %d"
--> 270                              % (X.shape[1], n_features))
    271
    272         scores = safe_sparse_dot(X, self.coef_.T,

ValueError: X has 2 features per sample; expecting 11

Here is the code:

#Split the variables
X = dataset.iloc[:, :11].values
y = dataset.iloc[:, -1].values

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Fitting Logistic Regression to the Training set
from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state = 0)
classifier.fit(X_train, y_train)

# Predicting the Test set results
y_pred = classifier.predict(X_test)

# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred)
acc_score = accuracy_score(y_test, y_pred)
print(acc_score*100)

Below is the visualization code:

# Visualising the Training set results
from matplotlib.colors import ListedColormap
X_set, y_set = X_train, y_train
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
             alpha = 0.75, cmap = ListedColormap(('red', 'green')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = j)
plt.title('Logistic Regression (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()

I know that the error is that the model has been trained using 11 functions, but it is envisaged to use 2 functions, but I do not know exactly what to change.


texts: How does keras calculate accuracy for multi label classification?

texts:
I am using this code for a multilabel problem classification.

from __future__ import print_function

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding
from keras.layers import LSTM
from keras.models import Model
from keras.datasets import imdb

max_features = len(vocabDic)
maxlen = 500  # cut texts after this number of words (among top max_features most common words)
batch_size = 32

train_set = sequence.pad_sequences(train_set, maxlen=maxlen)
test_set = sequence.pad_sequences(test_set, maxlen=maxlen)

print('train_set shape:', train_set.shape)
print('test_set shape:', test_set.shape)

print('Build model...')
model = Sequential()
model.add(Embedding(max_features, 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(90, activation='sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

print('Train...')
model.fit(train_set, train_labels,
          batch_size=batch_size,
          epochs=15,
          validation_data=(test_set, test_labels))
score, acc = model.evaluate(test_set, test_labels,
                            batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

My problem is there are 90 classes and the accuracy is too high from the second epoch. I suspect keras is computing something incorrectly.
Any clues?

Edit: I calculated the total recall of the model. It is barely 5%( 5 times better than random). Is it a normal behavior for such a problem?


texts: Let's say that our feature space is of dimension size = 9. Also, let's say that we apply LDA and we get only one LDA component. If we developed for example the SVM model and test the accuracy with these 9 features and then do the same with this one LDA component and calculate this new accuracy. IF the accuracy with the 9 features = the accuracy with this one LDA component, then this means that all the 9 features are important ?


texts: What algorithms are available to identify similarity scores between two records?

texts:
I have data that looks like this:

priority    task         area
1           clean room   living room
2           clean room   living room
2           water plants bedroom

I want to identify records with a priority of 2 that should be a priority of 1 based on similarity between the priority 2 task and each priority 1 task.

The result data would look like this:

priority    task         area         propensity_for_priority_1
1           clean room   living room  1
2           clean room   bedroom      .5
2           water plants bedroom      0

String similarity does not need to be performed (i.e. living room and bedroom are mutually exclusive). Is there a matching algorithm that can be used to compare various fields and result in one propensity_for_priority_1 score for each record?


texts: Does anyone have examples of using graph theory/network science in the field of Business Intelligence?

texts:
A lot of examples for the use of graph theory revolve around either social networks (which I understand are useful for business intelligence) or scientific/biological application.

I am exploring the application of graph theory on business data. The current applications that I have identified are:

Airport connections
Delivery logistics - like the one eBay
implemented
Co-purchasing networks
Website navigation
Bank transfers

Can anyone provide links to studies or companies already using graph theory in this field?


texts: In Keras, how to get 3D input and 3D output for LSTM layers

texts:
In my original setting, I got

X1 = (1200,40,1)
y1 = (1200,10)

Then, I work perfectly with my codes:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 1), return_sequences=True))
model.add(LSTM(12, return_sequences=True))
model.add(LSTM(6, return_sequences=False))
model.add((Dense(10)))

Now, I further got another time series data same sizes as X1 and y1. i.e.,

X2 = (1200,40,1)
y2 = (1200,40)

Now, I stack X1, X2 and y1, y2 as 3D arrays:

X_stack = (1200,40,2)
y_stack = (1200,40,2)

Then, I try to modify my keras code like:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 2), return_sequences=True))
model.add(LSTM(12, return_sequences=True))
model.add(LSTM(6, return_sequences=False))
model.add((Dense((10,2))))

I want my code work directly with the 3D arrays X_stack and y_stack without reshaping them as 2D arrays. Would you give me a hand on how to modify the settings? Thank you.


texts:
I am working for a recruitment company on developing machine learning algorithms to automatically classify job applicants as either to be interviewed or not be interveiwed. The data is highly imbalanced (i.e most applicants will not be interviewed.) Currently, to do this we use a machine learning model per job. We feed it examples of applicants with their status. However, this has generally low performance due to the small size of the training dataset.

A potential improvement on this approach could be creating a single model which takes in as input information on the applicant and information on requirements of the job and then predicting whether or not to interview. Is this approach likely to work and a technique that is used/has been researched in any context?

Note: the actual problem I am working on is to do with classifying publications for inclusion in medical systematic reviews. The 'different jobs' represent different review topics, the 'job requirements' are the criteria for inclusion and each 'applicant' is actually a medical publication. I used this example as it is comparable and requires less domain expertise to understand.


texts: Add Custom Labels to NLTK Information Extractor

texts:
I am working on an information extractor specifically purposed with parsing relationships between entities such as movies, directors, and actors. NLTK appears to provide the necessary tools to construct such a system. However, it is not clear how one would go about adding custom labels (e.g. actor, director, movie title).

Similarly, Chapter 7 of the NLTK Book discusses information extraction using a named entity recognizer, but it glosses over labeling details.

So, I have two questions:

How would I add custom labels?
If I have bare lists of relevant named entities (e.g. movies, actors, etc.), how can I include them as features? It appears that I would need to use IOB format, but I am unsure about how to do this when I only have lists of named entities.


texts: why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding

texts:
We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data.
And I real a blog which give the difference between Label Encoding and One-Hot Encoding.
So I am wondering why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order, $0 < 1 < 2$ if the data has been Label Encoding.
Does it have some provements and theories?
Or anyone just can explain it in intuition.
Thanks in advance.


texts: Can neural networks be considered a method for data analysis?

texts:
I am in the process of writing my thesis.
In this thesis I am stating that neural networks can be considered a method for data analysis.
After a bit of though I am now unsure if this claim is justified.
Wikipedia states:

"Data analysis is a process of inspecting, cleansing, transforming,
and modeling data with the goal of discovering useful information,
informing conclusions, and supporting decision-making."

So, a feedforward network does model the data, but because of its black-box nature it is problably less useful to discover useful information. However, because of its predicive capabilities it certainly could support decission-making. However, the main goal when using neural networks is still making plain predictions, not necessarily support decission making.
What is your take on this?
Can feedforward neural networks be considered a method for data analysis?


texts: For multi-class classification in SGDClassifier how do I tell if it is using one-vs-rest or one-vs-one by default?

texts: According to the Geron book, for multi-class classification, SGDClassifier in scikit-learn uses one-vs-rest. But how can I tell which one is used as it doesn't appear to give this information in the help file.


texts:
I have a large dataset "train" incl. a column ClientFreeSource from which I want to exclude an outliers. For that purpose I decide to use IQR score.
I've tried:

Q1 = train['ClientFreeSources'].quantile(0.25)
Q3 = train['ClientFreeSources'].quantile(0.75)
IQR = Q3 - Q1

train = train[~((train['ClientFreeSources'] < (Q1 - 1.5 * IQR)) |(train['ClientFreeSources'] > (Q3 + 1.5 * IQR))).any(axis=1)]

...which returns an error:

  ValueError: No axis named 1 for object type

I also tried to apply to score for no specific feature (whole dataset "train"), which returned:

  TypeError: Could not operate array([ nan]) with block values
  unorderable types: str() < float()

Help would be appreciated.
I'm new to DS.


texts: How to plot multiple bar charts with different infill and outline?

texts:
I have the CSV's of data. I want to plot the distribution of every attribute or a single one to compare the distribution from other CSV's, Since all attributes are the same but the distribution is different from each other because I have made the synthetic data using the original one.
Just like shown below that one attribute distribution is different from the other, I don't know the name of this chart. How could I plot this :

Code I tried so far:
    data=pd.read_csv(r'/content/drive/MyDrive/census-income.data.csv',names=['age','class of worker','detailed industry recode','detailed occupation recode','education','wage per hour','enroll in edu inst last wk','martial status','major industry code','major occupation code','race','hispanic origin','sex','member of labour union','reason for unemployment','full or part time employment stat','capital gains','capital losses','divdends from stock','tax filer status','region of previous residence','state of previous residence','detailed household and family stat','detailed household summary in household','instance weight','migration code-change in msa','migration code-change in reg','migration code-move within reg','live in this house 1 year ago',' migration prev res in sunbelt','num persons worked for employer','family members under 18','country of birth father','country of birth mother','country of birth self','citizenship','own business or self employed','fill inc questionnaire for veterans admin','veterans benefits','weeks worked in year','year','income'])
    data=data.drop(['state of previous residence','migration code-change in msa','migration code-change in reg','migration code-move within reg',' migration prev res in sunbelt','country of birth father','country of birth mother','country of birth self','instance weight'],axis=1)
    s_data=pd.read_csv('/content/drive/MyDrive/SyntheticData/SDV/SDV_TRAIN_DATA.csv')
    d_data=pd.read_csv('/content/drive/MyDrive/SyntheticData/Data Synthesizer/DataSynthesizerCensusTrain.csv')
    sp_data=pd.read_csv('/content/drive/MyDrive/SyntheticData/Synthpop/CensusTrainData.csv')
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd

    # example data
    x = np.arange(len(data))

    fig, ax = plt.subplots(figsize=(50,48))

    ax.bar(data['age'], x,color="red",  edgecolor="None", linewidth=4, label='Original Data')
    ax.bar(s_data['age'],x,color="None", edgecolor="blue", linewidth=4, label='Synthetic Data Vault')
    # ax.bar(d_data['age'],x,color="None", edgecolor="green", linewidth=4, label='Data Synthesizer')
    # ax.bar(x, sp_data, color="None", edgecolor="green", linewidth=2, label='Synthpop')
    ax.legend()

    plt.show()


texts: To calculate my confusion matrix with recall and precision, my test set need to be equal(balanced)?

texts:
In my CNN, I have 200 'negative' images and 50 'positive' images in my test set and I want to make a confusion matrix. My doubt is if I have to equalize the samples in the dataset because if I keep this 200 - 50 my precision falls because I have a lot of 'false positives'.

So, I have to divide the percentage of negatives by 50, or keep 200/50?

This is my results without balance the samples:

                    predicted positive         predicted negative
actual pos.              41                             9
actual neg.              31                            169

recall = 41 / 50 = 82%
precision = 41 / 72 = 57%


texts: I was confused when i was reading about weight pruning on CNN. Is it applied for all the layers including convolutional layers or only it is done for dense layers?


texts:
For multiclass problems, there are 2 versions of the AUROC metric: the AUROC OvR and AUROC OvO.
Does anyone know in what particular cases we would use AUROC OvR vs. AUROC OvO? In the general academic literature, is there a preference given to one?
Many thanks in advance!


texts: We are using PostgreSQL to store big data and are concerned it may crash the on-board Neural Net. Any thoughts?

texts: We have figured out how to write Big Data to our PostgreSQL database. We would like to run this (normalized) data into the on-board Neural Network in Orange. Our lead programmer said he thought Orange may very well choke on this large amount of data if we feed it directly into the Orange Neural Net from Big PostgreSQL Data. Does any one here have any experience with this problem? Should we maybe write a widget that caches the data and feeds it into the neural network in discrete chunks? Has anyone here attempted to feed gigabytes of data into the Orange Neural Net? Is TensorFlow a better solution?


texts: Python is reading my data with NANS and Infs, but they don't have any

texts:
I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using StandardScaler()because of the presence of Inf values or values too big for dtype(float) I checked in R (I am more comfortable with this language, but have to use Python for this project) and it shows that the dataframe does not have any Inf values or NAs.

Are there restrictions in Python that converts large numbers or small numbers to NA/NaN/Inf?

Thanks.


texts: Preserving explained variance while reducing dimensionality

texts:
We have a function $f:R^N \rightarrow R$ and a set of points $D=\{ x\in R^N\}$. How is it possible to linearly lower the dimension of points to $M \ll N$ such that the fraction of explained variance* will be preserved as much as possible?

we refer to the fraction of explained variance in prediction of output of $f$ using dataset $D$


texts: How are Lagrangian multipliers zero except for support vectors in dual representation of SVM?

texts:
How can we conclude that the Lagrangian multipliers are zero, except support vectors, in a dual problem? I cannot seem to see it.

$$L(\alpha)=-\frac{1}{2}\sum_i \sum_j \alpha_i \alpha_j y_i y_j x_i' x_j + \sum_i{\alpha_i}
$$


texts: how to export the data edited to a new excel file

texts:
The provided code imports the excel file and removes the not NaN values and store it in df2. Now, How do i save the changes to an another file.

df1 = pd.read_excel('file_name.xlsx')
df2 = df1[pd.isnull(df1['column_name'])]
df2


texts:
Have been learning Machine Learning concepts and doing hands-on past few months. Now got a bit different doubt than implementation of code.

This is regarding measures to be considered while deploying in a Prod environment, say something like "A newbie's checklist to ML production" example -

Are Feedback loops always considered?
Is cross-validation done in final prod model or only during train/test the model?
How the auto-suggestion in Amazon or any ecommerce website works? - Meaning does the model get trained on every page load or already pre-trained and just predicts (suggests) based on user selections?

I know the questions are too general and newbie, but will help many to understand the concepts relating to real-world. Please advise.

Let me know for any further information and I will be happy to furnish/respond.


texts: How do I represent a hidden markov model in data structure?

texts:
My task involves a POS Tagging using HMM. I am given a training data set (word/tag). I have to write a file with transition probabilities and emission probabilities. I am currently using a nested dictionary of the form {State1: {State2: count, State3 :count}}. However, while calculating the probabilities now via the counts in nested dict, my program is running very slow for mid size files (e.g. 2000 sentences and tags)

Is there a better way to store a HMM in python? For my project, I cannot use any external library that already does this, I must use standard python libraries.


texts: Is it my Sales growth is depend on Commisions/Discounts?

texts:
I prefer this model in R

We are capturing sales data by time series (Month by month). Some of items have commissions and some have Discounts and others have both commissions and discounts. Is it Commissions or Discounts or commissions + Discounts have impact on my sales growth? Or is it my sales are growing because of those commissions or discounts or discounts +commissions Can you suggest me best model to solve my use case? I am thinking multiple regression. But I want to double check with experts like you.

Thanks for your all your help

Sample Data set: (5 variables)

Year-Month -Product -Sales  -Commission -Discounts
2013-01 Milk    300 No  Yes
2013-02 Milk    400 No  Yes
2013-03 Milk    200 No  Yes
2013-04 Milk    150 No  Yes
2013-05 Milk    500 No  Yes
2013-01 Bread   800 Yes No
2013-02 Bread   879 Yes No
2013-03 Bread   790 Yes No
2013-04 Bread   459 Yes No
2013-05 Bread   600 Yes No
2013-01 Cheese  400 Yes Yes
2013-02 Cheese  350 Yes Yes
2013-03 Cheese  600 Yes Yes
2013-04 Cheese  590 Yes Yes
2013-05 Cheese  720 Yes Yes


texts: What is the optimal method to evaluate clusters further?

texts:
Supervised learning is straightforward on medical data using Orange, but unsupervised learning is more challenging. I selected a data set based on Florida County Health Ratings where individuals rated their health based on about 40 variables, from 67 counties. (I left the county names out). The data is continuous which is good. I added the preprocess widget so I could impute missing data and normalize the data. I used K-means clustering and with 2 clusters (C1,C2) the silhouette score was 0.223. I then connected the silhouette plot widget to a scatter plot and a data table. In the data table, most of the silhouette scores were 0.5-0.6. The scatter plot showed fairly good separation. On the X axis is the percent_fair/poor health rating and on the Y axis is physically unhealthy days per week. Clearly, it is separating individuals into healthy and unhealthy clusters. I opted to use the save data widget and export the clusters to a CSV file. I have done simple averages of other variables like family income and obesity and once again, C1 reflects healthy individuals and C2 reflects unhealthy individuals. I simply want to know is that how you would recommend analyzing the clusters? I also realize that I can convert the output of the two clusters into a classification model and evaluate the clusters that way.
I will cut and paste my workflow and my scatterplot below. Thanks.


texts: What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?

texts:
I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to send to each of featuresCol, labelCol, predictionCol, and probabilityCol.

From the docs I gather that:

featuresCol is a list of features from your dataframe
labelCol is the target feature
predictionCol is also the target feature, but generated by the model (not sure). Do I need to set this prior to training?
probabilityCol is the probability of each class as a vector. Is this similar to sklearn's class_weight? ie does the model account for a low diversity? If so how?

Additionally, can I set an option for OOB_score?

clf = RandomForestClassifier(featuresCol=feature_cols, labelCol=target_col, numTrees=300, MaxDepth=15, Impurity='gini', maxMemoryInMB=2**10)
clf_t = clf.fit(train)

y_train_pred = clf_t.transform(test)
y_test_pred = clf_t.transform(test)

Here's a link to the docs: https://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#module-pyspark.ml.classification


texts: Deep learning error: All layer names should be unique

texts:
I am working in deep learning project using vgg16.
I got the following error and I could not solve it.

RuntimeError: ('The name "block1_conv1" is used 2 times in the model. All layer names should be unique.

The code:

vgg16_model = keras.applications.vgg16.VGG16()

input_layer1 = vgg16_model.input
vgg16_model.get_layer(index = 0).name = 'input1'
last_layer1 = vgg16_model.get_layer('flatten').output

# Adding layers
x = Dense(2048, activation='relu', name='fc1')(last_layer1)
x = BatchNormalization(name='bn1')(x)
fc12 = Dense(2048, activation='relu', name='fc2')(x)
bn12 = BatchNormalization(name='bn2')(fc12)
out1 = Dropout(0.4, name='dropout1')(bn12)

#Classification layer
output_layer1 = Dense(no_classes, activation='softmax', name='prediction1')(out1) #106 classes
model1 = Model(input_layer1, output_layer1)

#Train the last block
model1.trainable = True
set_trainable = False
for layer in model1.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

vgg16_model2 = keras.applications.vgg16.VGG16()

input_layer2 = vgg16_model2.input
vgg16_model2.get_layer(index = 0).name = 'input2'

# Remove layers from block4_pool layer
last_layer2 = vgg16_model2.get_layer('block4_pool').output

# Adding layers
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv1', padding='same')(last_layer2)
x = BatchNormalization(name='bn1')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv2', padding='same')(x)
x = BatchNormalization(name='bn2')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv3', padding='same')(x)
x = BatchNormalization(name='bn3')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name = 'block5_pool')(x)
x = Flatten(name = 'flatten')(x)
x = Dense(2048, activation='relu', name='fc1')(x)
x = BatchNormalization(name='bn4')(x)
fc22 = Dense(2048, activation='relu', name='fc2' )(x)
bn25 = BatchNormalization(name='bn5')(fc22)
out2 = Dropout(0.3, name='dropout1')(bn25)

#Classification layer
output_layer2 = Dense(106, activation='softmax', name='prediction2')(out2)
model2 = Model(input_layer2, output_layer2)

#Train the last block
model2.trainable = True
set_trainable = False
for layer in model2.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

con = concatenate([out1, out2])  # merge the outputs of the two models
output_layer3 = Dense(no_classes, activation='softmax', name='prediction3')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

I tested model1 and model2, and I got good results. But when I tried to merge the two models the error appear. The problem initiated here:

multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

Please help to solve this problem.


texts: What is the difference between Okapi bm25 and NMSLIB?

texts:
I was trying to make a search system and then I got to know about Okapi bm25 which is a ranking function like tf-idf. You can make an index of your corpus and later retrieve documents similar to your query.
I imported a python library rank_bm25 and created a search system and the results were satisfying.
Then I saw something called Non-metric space library. I understood that its a similarity search library much like kNN algorithm.
I saw an example where a guy was trying to make a smart search system using nmslib. He did the following things:-

tokenized the documents
pass the tokens into fastText model to create word vectors
then combined those word vectors with bm25 weights
then passed the combination into nmslib
performed the search.

If the above link does not opens the document just open it in incognito mode.
It was quite fast, but the results were not satisfying, I mean even if I was copy pasting any exact query from the doc, it was not returning that doc. But the search system that I made using rank_bm25 was giving great results. So the conclusion was
bm25 gave good results and nmslib gave faster results.
My questions are

How do they both (bm25, nmslib) differ?
How can I pass bm25 weights to nmslib to create a better and faster search engine?
In short, how can I combine the goodness of both bm25 and nmslib?


texts: Amalgamating multiple datasets with different variables coding

texts:
I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables and make them consistent?

If there is such thing, that would save me months of tedious work. The data is stored in SPSS format.


texts: How to Determine Specific Activation Function from keras' .summary()


texts: Is there a library or algorithm to identify forgery in images of documents?

texts:
I am trying to implement a deep learning solution to identify if there is any forgery in images of documents. This solution for example, https://github.com/ISICV/ManTraNet, is capable of identifying forgery in images of people or objects, however I have tried it with images of documents and it fails to do the job with such level of accuracy. For a better idea of what I am trying to achieve see the example below. Mind that this example has been constructed for explanation purposes only.
This is a dummy check:

I have forged this check by changing the check number at the top right, the name, and the first part of the number at the bottom. See below:

I would like to develop a solution that can classify the image above as forged, taking into account that often times indications of forgery are very subtle and that real examples are images of these documents and are not digitally created such as the ones above. Is there any paper, algorithm, or library that can help in this case? I have looked at multiple forgery detection solutions and papers and tried implementing a couple of CNNs with transfer learning, however they fail at getting this task right with robust results.


texts:
I would like to develop a soccer field segmentation method. For this purpose, I prepared a training image data set and annotated field and non-field pixels. Following is a gr-chromacity plot of all training samples, colored with respect to their labels.

I want to train a classifier for inferring the label of a new sample. The first approach comes to my mind is using Gaussian mixture models to model both distributions. Would you recommend another method for this purpose?


texts:
I have a regression problem whereby my data has 21 features and I wish to apply dimensionality reduction using PCA. As far as I know, all the tutorials I have seen so far use PCA for classification problems. I did do PCA for regression but I am un-able to display the nice scatterplots that show the PC1 on the x-axis, the PC2 on the y-axis, and the targets in the middle.

I wrote the following code

        X = self.X

        pca = PCA(n_components=NUM_FEATURES_PCA)
        principal_components = pca.fit_transform(X)

        principalDf = pd.DataFrame(data=principal_components,
                                   columns=['PC1', 'PC2'])
        finalDf = pd.concat([principalDf, self.df[[self.target_variable]]], axis=1)
        plt.scatter(finalDf.loc[self.df[self.target_variable], 'PC1']
                   , finalDf.loc[self.df[self.target_variable], 'PC2'], s=50)
        plt.xlabel('PC1', fontsize=15)
        plt.ylabel('PC2', fontsize=15)
        plt.title('2 component PCA', fontsize=20)
        plt.show()

So in other terms, could we display such plots for PCA in regression ? Or should we transform the continuous target variable to a categorical (labeled one) through binning or like-wise ?

references: these plots


texts: When to use BayesianSearchCV and how it works?

texts:
Can somebody highlight when to use BayesianSearchCV and how it works? I have seen the implementation of same on kaggle and wanted to explore it further.

Below is the link where the implementation has been performed.

Kaggle link to Bayesian Optimization


texts:
I have been trying to plot my data from a cvs file with Bokeh but I couldn't find a smoothing option to smooth my graph curve.

Is there any in bokeh.plotting for this task? If not, how else should I proceed?


texts: Random Forests with Big Data - number of trees v. number of observations

texts:
I frequently use Random Forest, Regularized Random Forest, Guided Random Forest, and similar tree models.

The size of the data that I'm dealing with has grown beyond what I can work around using HPC and parallelism. It's typically large due to row length (observations) not columns (features). The data is also often not normally distributed.

I have to make a choice between:

Running a small number of trees (i.e. 50 or less) with either complete data or a relatively large and comparative sample
Running several times the number of trees, but with a correspondingly scaled down sample size

There are work-arounds and for any 1 case -- for instance, I can do some ad hoc tests to see which I think will work better, but what I'm wondering is if there is a good theoretical (or robust empirical) reasoning to either guide the choice of approach over the other or to describe the tradeoff being made?

In other words, I'm hoping that someone more comfortable with the math, statistics, and theory underlying this (type of) algorithm can offer some generalizable insight.


texts:
I am using Random Forest model for feature selection on data with bias. I have tried the Random Forest model on my features by increasing number of trees and found the feature importance moving.
Could anyone help me understand how I should use the 'rfpimp' package in this situation in Orange visual programming?


texts:
Can anyone help me with this error. I did the following code but it does not work and I am getting the following error:

ValueError: Expected 2D array, got scalar array instead:
array=6.5. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

My code:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import pandas
dataset = pandas.read_excel('PEG RATIOS.xlsx')

X = dataset.iloc[:, 2].values
X =X.reshape(-1,1)
y = dataset.iloc[:, 3].values
y = y.reshape (-1,1)

from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(X, y)

from sklearn.preprocessing import PolynomialFeatures
poly_reg = PolynomialFeatures(degree = 4)
X_poly = poly_reg.fit_transform(X)
poly_reg.fit(X_poly, y)
lin_reg_2 = LinearRegression()
lin_reg_2.fit(X_poly, y)

X_grid = np.arange(min(X), max(X), 0.1)
X_grid = X_grid.reshape((len(X_grid), 1))
plt.scatter(X, y, color = 'red')
plt.plot(X_grid, lin_reg_2.predict(poly_reg.fit_transform(X_grid)), color = 'blue')
plt.title('PEG Ratios verrus Exoected Growth: Semiconductor Firms')
plt.xlabel('Expected Growth rate')
plt.ylabel('PEGH Ratio')
plt.show()
lin_reg_2.predict(poly_reg.fit_transform(6.5))


texts: Prediction with multiple features (attributes)

texts:
I recently used a prediction algorithm to try to predict how much an user is willing to interact with an item using ALS (Matrix Factorisation) but I had to "condense" multiple attributes into one, deciding some arbitrary weights for the values to be able to calculate the rating to use.

It works (sort of) but I think it is definitely wrong because weighting the attributes manually is static and do not adapt to the data themselves dynamically.

I was thinking to switch to a Distributed Random Forest algorithm but even if it would use all the attributes (randomly) they would not be used all together and this makes it not fit for my purposes.

During my researches I found out that the Recurrent neural networks may be fit for what I am trying to achieve but before focusing on them, because I am not really an expert in the field, I would like to have suggestions :)

Potentially I could take advantage of the time-series feature of the RNNs but for now I prefer to focus on supporting the set of attributes I have.

Currently the "rating" for the ALS is calculated weighting some attributes like:

how many times the user viewed the item page
how many times the user viewed the category page
how many times the user viewed the sub category page
how many times the user bought something from the category
how many times the user bought something from the sub category
how much time ago the user bought the item
how much time ago the user bought something from the same category
how much time ago the user bought something from the same sub category

(and similar)

Is it worth trying with an RNN? If yes, should I try to go with tensorflow or caffe?

Thank you!


texts: Is it a best practice to exclude retweets from the data set?

texts:
I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is better to remove all the retweets?

Thank you,


texts: What are the best models developed for image classification

texts: For inventory detection problem, when you cut single product images from the shelf, the next step you have to do is that you should label the product and identify which category this product belongs to. In this, step you have a classification problem with thousands of classes and a few images per each class for training(maybe between 10 to 20 pictures or maybe less for some of the specific products). My question is, is there any model developed for this purpose? In which you can train your classification model with thousands of categories and a few image for each category


texts: What's the issue with my code for visualizing linear regression in 3 dimensions with matplotlib?

texts:
I am trying to use linear regression that takes two variables "Idade" and "LF" and tries to predict a third one, "DGAF". I'm trying to both do the scatterplot with the observations and the model prediction on the same graph. Bellow is the code I used (using Python).
X = df[["Idade","LF"]]
y = df["DGAF"].values.reshape(len(df["DGAF"]),1)

reg = LinearRegression()
reg.fit(X, y)

fig = plt.figure()
ax = Axes3D(fig)

ax.scatter(X['Idade'],X['LF'], y,s=10)

x_pred = np.linspace(0,100,1000)
y_pred = np.linspace(0,100,1000)
xx_pred, yy_pred = np.meshgrid(x_pred, y_pred)

model_viz = np.array([xx_pred.flatten(), yy_pred.flatten()]).T

ax.set_xlabel('Years since oil change')
ax.set_ylabel('LF')
ax.set_zlabel('DGA Score')

ax.plot(model_viz[0], model_viz[1], reg.predict(model_viz),color='red')
plt.show()

I get this error:
ValueError: input operand has more dimensions than allowed by the axis remapping

I am able to do the scatterplot, the issue seems to be with plotting the prediction. How can I solve this?


texts: XGBOOST (sklearn interface) REGRESSION error

texts:
I am trying to run a GRIDSEARCHCV (sklearn) on XGBRegressor. Documentation on the parameter says that if regression, then objective = reg:squarederror.(see https://github.com/dmlc/xgboost/tree/master/demo/regression) However, whenever I am trying to run the search, I am getting an error saying XGBoostError: b'[13:39:54] src/objective/objective.cc:23: Unknown objective function reg:squarederror.

I am not sure how to get around this problem. For the sake of completeness, below is the piece of code I am using for this purpose.

cv_params = {
             'n_estimators' : np.arange(100, 1201, 100),
             'max_depth' : np.arange(2, 10)
             }

xgbr_params = {'objective':'reg:squarederror','n_jobs':-1,'random_state':4444,'min_child_weight':1,
              'eta':0.3,'subsample':0.8,'gamma':0.5,'colsample_bytree':0.8}

opt_xgbr = GridSearchCV(xgb.XGBRegressor(**xgbr_params)
                              ,param_grid=cv_params,scoring='r2',cv=5,n_jobs=-1,return_train_score=True, verbose=3)

Any help would be greatly appreciated.

Thanks