texts: Binary classification: is there a tradeoff between predicting class 1 vs 0?

texts:
I have been digging much more in detail into classification performance metrics lately to get my head around the 'dynamics' of classification algorithms. What I have noticed is that in binary classification problems, I have so far NOT seen an algorithm that does well predicting both classes at the same time.

To me, an algorithm is considered having an 'edge' for a given class if the accuracy on that class is higher than the proportion of that class in the dataset. E.g : if we have 20% Class A and 80% Class B, I would consider the model being 'performant' if accuracy on class A is >20%, or if accuracy on Class B is >80%.

The actual performance metric I calculate at in practice are: (% accuracy Class A-20%) and (% accuracy Class B-80%) for the example of a 20/80 distribution.(% accuracy Class A)=(# corrrecly predicted points A/# points A).
Any value of this metric above 0 means the model has an 'edge' predicting that class.

However, what I have noticed is that it is extremely rare that I find values above 0 for both classes. Most of the time (if not always), when there is an edge in a class, there is a somewhat equal or larger loss in the other class.

Formally, taking the sum of (% accuracy of Class A minus 20%)+(% accuracy of Class B minus 80%) results usually in 0 or less. This means you 'gain power' predicting a class, but you lose that power, or more, for the other class. Overall I like this metric because it beats accuracy in case of a majority class 'vote' for a biased model. In case of the majority class bias, this metric would return 0 (try it out you will see, it will end up being -20%+20%=0)

Anything above 0 would be a good model to me. I then thought meta ensembles are the natural progression of this, where u 'just' need to combine the powers of models that have an edge on each class. Hopefully the meta-ensemble will have a value above 0 for all classes for my made up metric.

Am I misinterpreting/understanding anythere here? Is this normal behaviour for binary classifications? Is it not possible to be able to predict both classes above their distribution in the dataset?

Thanks a lot for your enlightments.

P.S: In scikit-learn i always use class_weight='balanced' to avoid balancing/majority class issues etc.


texts: Strategies to encode categorical variables with many categories

texts:
I was going over the Kaggle competitions IEEE,Categorical Feature Encoding Challenge and one of the ways in which categorical variables have been handled is by replacing the variables by the respective count frequencies of the variables in the column. I understand why we need to transform the variables, but can someone please explain why this approach was used. What exactly is the logic behind it?

Secondly, I was wondering if anyone may be willing to share other techniques of dealing with categorical variables (excluding the obvious one hot encoding) or point me some interesting approaches for the same. Thank you!


texts: python: why add new features of image to be worse result with xgboost

texts:
there are about 93 rows of data with features, two classes label. And, there are about 49 one-hot value features, and there are about 10 features continuous value. I split the data randomly by train and valid data to predict it with following code:

model = XGBClassifier()
model.fit(X_train2, y_train2)
y_pred = model.predict(X_valid)
accuracy = accuracy_score(y_valid, y_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))

I found that when I just used the one-hot value features, the accuracy is 78.95%, but when added another 10 features with continuous values, and the accuracy is reduced sharply, and is 57.89%.More, based on the feature importance, the ranking top 5 features is the continuous values features, also other continuous features have importance value. But most of the one-hot value feature importance value is 0, just 7 features have value. Of course, just use the one-hot value, there are just about 17 features have value, others are 0.

In addition, just used the 10 continuous features, the accuracy is 68.42%.

I don't know clearly why caused such kind of result.
now I guessed that one of reason may be the number of data is not so big, and the features is almost about 59 features.


texts: Transform multi-label problem to multi-class problem

texts:
What are the downsides of modelling a multi-label problem as a multi-class problem with a single classifier?

Let my clarify what I mean.

There at least two ways that one multi-label problem can be transformed to a multi-class problem with a single classifier (suppose there are N labels at our problem):

1)

Create a class for each element of the powerset of the labels.

Therefore, each element for each combination of the labels.

In this case, the output vector will have $2^N$ length.

2)

Have an output vector of $N$ length (every element of the vector will be one label) but the problem will be treated as a multiclass problem with one classifier.

In this case, let's say the classes which have output probability more than 0.2 will be considered as the classes/labels of this instance/observation.

Obviously, the output probabilities of all the classes should sum up to 1.

What are the implications of transforming a multi-label problem to multi-class problem at each of these cases?


texts: How to reverse engineer a logarithmic equation

texts:
I am trying to reverse engineer the parameters of a human-designed logarithmic equation. Here are the facts:

The equation is of the type a = x * ( y ^ b )

a and b are known, x and y are unknown and need to be determined

I have a table of values giving b as an exact value and a as the rounded integer result. For example, b might be 7 and a might be 68 where the 68 is a value rounded to the nearest integer. The number of values in the table is around 12-15 or so.

x will be some number which is human selected, so it will be a relatively round number to +/- 0.05. So, for example, x might be 53 or 52.75, but it would not be 51.1938434.

y is some positive rational value like "1.093534".

So, given the table of values of a and b and the facts above, how can I do an estimation to determine the most likely values of x and y?


texts: Python sample function TypeError: '<' not supported between instances of 'str' and 'int'

texts:
i GOT this ERROR when doing sampling stratify of a sample dataframe
code is like this :

df = pd.DataFrame(dict(
    A=['a', 'a', 'a', 'b', 'c', 'c', 'c', 'a', 'b', 'b', 'b', 'c', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'],
    B=range(20)

 ))

df.sample(n=100, weights='A',replace=True)

it gives me this error:

  TypeError: '<' not supported between instances of 'str' and 'int'

anyone can help?


texts: How to decrease the learning rate only when cost function is stagnant?

texts: I'm training a very complex function in tensorflow. Is there a way to decrease the learning rate only when the cost isin't decreasing very much?


texts: Combine Pandas DataFrame Rows Based on Matching Data and Boolean

texts: I have a Pandas DataFrame with sales data and columns for year, ISO week, price, quantity, and organic [boolean]. Because each row represents a different location, dates are repeated. I would like to combine rows with matching year, ISO week, and organic. Ideally, the combined row would have the average price and sum of total quantity. Any help is much appreciated!


texts: Will a Count vectorizer ever perform (slightly) better than tf-idf?

texts:
For the task of binary classification, I have a small data-set of a total 1000 texts (~590 positive and ~401 negative instances). With a training set of 800 and test set of 200, I get a (slightly) better accuracy for count vectorizer compared to the tf-idf.

Additionally, count vectorizer picks out the relevant "words" training the model, while tf-idf does not pick those relevant words out. Even the confusion matrix for count vectorizer shows marginally better numbers compared to tf-idf.

TFIDF confusion matrix
[[ 80  11]
 [  6 103]]
BoW confusion matrix
[[ 81  10]
 [  6 103]]

I haven't tried cross-validation yet though it came to me as shock that count vectorizer performed a bit better than tfidf. Is it because my data set is too small or if I have't used any dimensionality reduction to reduce the number of words taken into account by both the classifiers. What is it that I am doing wrong?

I am sorry, if it is an immature question, but I am really new to ML.


texts: Link between Correspondance Analysis and MCA

texts:
I've been dealing with a dataset full of categorical variables and had some issues to apply dimensionality reduction techniques as PCA is no great fit for such datasets.

After some research, I came across Multi Correspondence Aanalysis for the first time and read several documents about it, mainly this one (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.470.7630&rep=rep1&type=pdf) and they all talked about CA (Correspondence Analysis) since MCA is basically CA but when you have more than only two variables so I've been trying to read about that too.

The problem for me is when they say "MCA is simply CA applied to the indicator matrix"; I really can't understnd that statement. I understand it when they say that MCA is CA applied to the Burt Matrix and it makes sens for me, but when it's stated that MCA = CA + indicator matrix (see pages 3-4 in the document above for example), I can't see it.

Could anyone shed some light on that please ?

Thanks !


texts: How to bin Colour values (categorical- high cardinality) to primary colours?

texts:
I have a colour variable with values for different e-commerce products like - olive, teal, blue, green etc.

I want to bin these colours to the primary colour values.

eg -

olive ---> green
teal ----> blue
blue and green will remain the same.

I also have images available of these products, therefore, I am wondering if there is any solution using some clustering technique or using a pre-trained model for classification in some way.


texts: Does it exist feature selection/reduction techniques in $O(n \cdot d)$?

texts:
I'm curious to know if feature selection and/or feature reduction techniques exist, which are linear on number of data $n$ and on number of dimensions $d$.

References and source code are very welcome.


texts:
I'm new to NER and I've been trying to extract names using Spacy. Here's my code:
import spacy
spacy_nlp = spacy.load('en_core_web_sm')

doc = spacy_nlp(text.strip())

# create sets to hold words
named_entities = set()
money_entities = set()
organization_entities = set()
location_entities = set()
time_indicator_entities = set()

for i in doc.ents:
    entry = str(i.lemma_).lower()
    text = text.replace(str(i).lower(), "")
    # Time indicator entities detection
    if i.label_ in ["TIM", "DATE"]:
        time_indicator_entities.add(entry)
    # money value entities detection
    elif i.label_ in ["MONEY"]:
        money_entities.add(entry)
    # organization entities detection
    elif i.label_ in ["ORG"]:
        organization_entities.add(entry)
    # Geographical and Geographical entities detection
    elif i.label_ in ["GPE", "GEO"]:
        location_entities.add(entry)
    # extract artifacts, events and natural phenomenon from text
    elif i.label_ in ["ART", "EVE", "NAT", "PERSON"]:
        named_entities.add(entry.title())

The model seems to have a decent accuracy with certain kinds of names. However it is unaware of how people’s names can differ around the world (not adapted to suit cultural differences). Is there a possible workaround to avoid this bias?


texts: Clustering : handling categorical data, should we pivot and scale?

texts:
I'm a SQLServer DBA and in the new version of this tool, there are new features to integrate R scripts and use it easily with the DB objects. That sounds cool.

But to use that, we have to know a little bit about stats and data mining.
And it's quite hard to find friendly and readable documentation for newbies like me. So, I will ask for some best-practice about clustering.

I have a dataset with few hundreds records and with 5 features, both categorical and numerical.

Regarding categorical data, I have a few basic questions :

how can I manage categorical data ? what I understood from articles found is that I can transform categorical data in boolean and "pivoting" it. If a have a column with categorical data (for instance with values 'a' or 'b'), creating to new columns like column_a and column_b with value 0 or 1 ?
should I always center / scale this new column ?


texts: How can I create .nii (nifti) file from 3D Numpy array

texts: I have a prediction numpy array. How can I make a .nii or .nii.gz mask file from the array?


texts:
Based on the library featexp I am trying to identify noisy categorical features. I want to know if this is the right way and if there is some library for this solution it will be great to know it.
I am using now this:
fig = plt.figure(figsize = (15,10))
tab = pd.crosstab(data_train['device'], data_train['hasSale'],normalize='index')
print(tab)
ax1 = fig.add_subplot(2,2,1)
tab.div(tab.sum(1).astype(float), axis=0).plot(kind="bar", stacked=True,color=["#3f3e6fd1", "#85c6a9"],ax=ax1)
plt.title('Train Data')
plt.xlabel('Device')
plt.ylabel('Percentage')

tab = pd.crosstab(data_test['device'], data_test['hasSale'],normalize='index')
print(tab)
ax2 = fig.add_subplot(2,2,2)
tab.div(tab.sum(1).astype(float), axis=0).plot(kind="bar", stacked=True,color=["#3f3e6fd1", "#85c6a9"],ax=ax2)
plt.title('Test Data')
plt.xlabel('Device')
plt.ylabel('Percentage')

The result:

As we can see, the distribution of the values is completely different in the train and the test data set.

How to remove the noise?
How can I calculate the correlation train trend?
Is this a good way to detect noise or there is some better way?


texts: Model with rank as the dependent variable?

texts:
I have a dataset with n records. The input features are f1, f2, f3 ... fm, and the dependent variable (output) is just the rank (not any type of scores).

What type of model should I build so that I can use the model to predict the rank of a new dataset? I suppose I could build a linear regression by assuming the dependent variable is just the rank (i.e. score = rank), but I am wondering if there is a better model or approach for this problem. Thanks!


texts: How to filter out a portion of the text data with re.match or any other regex method with Python?

texts:
I have the following data :

Node : WATER-7609-WR1

IP Address : 172.20.150.81

Severity : 5 - Critical

ServerSerial : 50505223

SNSupportGroup : UC-WAN

Node Location : CO 160 WATER 8 FL RCV

Event Details : Unknown Not Link Down or Up trap on Serial Channel:  Unknown Severity: 4
NOTE:Configuration Item is not found in CMDB. Please update Incident with related Configuration Item

I am interested in all text after "Event Details :",
 I have used the the following function :

def eventdetails(text):
   match =re.search(r'Event Details :[^.]*',text)
   return match

print(eventdetails(dfnetcool_inc["Description"][3]))

output: <_sre.SRE_Match object; span=(159, 292), match='Event Details : Unknown Not Link Down or Up trap >

--> Am not getting all of the text after Event Details. Perhaps there is an EOL charecter - how do I get remainder of the text, request your help, TIA!


texts: How to classify a dataset into 5 classes even though the performance is low?

texts:
I have a dataset of 5 classes with 8 features, say A, B, C, D and E. Now when I try to classify these into individual classes, I get accuracy, specificity and sensitivity of approx 50-60%, which is not a good rate. Now since I made the dataset myself, I know that classes [A,B,C,D] can be categorized into 1 class and [E] into another. Once a sample has been predicted to be in class [A,B,C,D], I can again break the class distribution to [A,B,C] and [D]. Once a sample has been predicted to be in class [A,B,C], I can again break the class distribution to 3 final classes [A],[B] and [C]. I pass my dataset by combining multiple classes into one as mentioned above to get the output. In the first stage of classification I get to know whether a sample is class E or not, in the second I get to know whether the sample (if not belonging to E) belongs to D or not. In the third I get to know whether a sample which is not E and D belongs to A or B or C.

Is this method of pipelining viable and a logical solution?


texts: If a dataset is imbalanced in real life, should I train on my machine learning model on an imbalanced dataset

texts: I have a dataset where around 20% of the data is the positive class and 80% of the data is the negative class. When I undersample and train my classifier on a balanced dataset and test on a balanced dataset, the results are pretty ok. However, if I train on the balanced dataset and test on an imbalanced dataset that replicates the real world (80-20 split) the metrics are not great. Should I train the model on the original imbalanced dataset if I want it to perform well on real world test data that is also imbalanced.


texts: Text classification into thousands of classes

texts: Could somebody point me to a paper or code that is about classifying texts into potentially thousands of categories (topics)? I do have data based on Wikipedia and the number of categories is really big (thousands), so looking for some solutions that should work.


texts: Sigmoid activation function for scaled continuous data

texts:
I've been working on a NLP project that attempt to output a single numeric value. The natural form of the data is integers between 0 and 27, with 27 being an absolute maximum, and values above 27 being meaningless.
I've used a RELU activation function for my output node as this seems to, at first glance, make most sense. However, this does (occasionally) lead to values higher than 27. I don't find this particularly problematic, because I can just post-hoc change all values >27 to 27. However, it has been suggested that I scale the data to be bound between 0 and 1 so I could use a sigmoid function as an activation function. However, I'm skeptical that this makes sense given the data. My data is linear, while the sigmoid function exponentially approaches both 0 and 1 with increasing difficulty, and is generally only used when fitting Boolean data. However, this is only an intuition, and I can't really support my hypothesis mathematically. Is there any reason a sigmoid function is or isn't a good fit for this data?
Bonus: anyone every implemented with Keras (R)? I've had issues even attempting to train a sigmoid function with non-Boolean data.


texts: Adding a Dense Layer to a Keras Model when using bert-for-tf2

texts:
I am learning how to apply BERT to a Keras model by following this tutorial. The model they create is slightly different in architecture than the Keras model I have worked with thus far as they are using this library (bert-for-tf2) to do it.
The Model:
from tensorflow.keras.layers import  Input
input_word_ids = Input(shape=(MAX_LEN,), dtype=tf.int32, name="input_word_ids")
input_mask = Input(shape=(MAX_LEN,), dtype=tf.int32, name="input_mask")
segment_ids = Input(shape=(MAX_LEN,), dtype=tf.int32, name="segment_ids")

# intilize model
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam

#  output
from tensorflow.keras.layers import Dense
pooled_output, sequence_output = bert_layer([input_word_ids, input_mask, segment_ids])

clf_output = sequence_output[:, 0, :]
output = Dense(1, activation='sigmoid')(clf_output)

model = Model(inputs=[input_word_ids, input_mask, segment_ids],
              outputs=output)
model.compile(Adam(lr=2e-6), loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()

# train
train_history_2 = model.fit(
    train_input, train_labels_enc,
    validation_split=0.2,
    epochs=2,
    batch_size=32
)

model_2.save('model.h5')

My Question:
My question is how can I add additional Dense layers between the Input and Output? This can either be done using the current model, or using the Sequential Class.
What I have tried:

I tried adding layers and connecting them to the input and output layers using the current architecture, but I cannot figure out how to make a "correct" connection.

I tried rewriting the model using the Sequential Class using the following template, but once again, I cannot figure out how to "correctly" connect the input to the layers, and then to the output.

    model = keras.Sequential(
        [
            layers.Dense(2, activation="relu", name="layer1"),
            layers.Dense(3, activation="relu", name="layer2"),
            layers.Dense(4, name="layer3"),
        ]
    )
    x = tf.ones((3, 3))
    y = model(x)

Any guidance would be highly appreciated. Thank you!


texts: What are some standard ways of computing the distance between documents?

texts: When I say "document", I have in mind web pages like Wikipedia articles and news stories.  I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics, with stronger preference for the latter.


texts: Design language or artifact for data science model

texts:
Inspired from this question, given that there is no standard definition of a data-science model, is there a way to standardize the communication about any data-science model?

Just like material design is a design language to communicate a UI design, is there a similar design language to talk about data-science models as well?

Or any other alternative or industry standard artifacts that communicates intent, purpose, approach and design of data-science model?


texts: How to plot calibration curve for multi-class problems?

texts:
How to plot calibration curve for multi-class problems, for example the available example on python plots it for 2 classes but here in the e-book link it is done for multi-classbook

how can I do so in python or R?


texts: Efficient way to cluster millions of face embeddings

texts:
I'm currently working on a face clustering system that gets incremental new input data.
I use the 128d face embeddings given by FaceNet. I already tested these Algorithms: Chinese Whispers, DBSCAN and HDBSCAN (doesn't seem to work at all for high dimensional data with over 50000 datapoints).
All of them take a very long time if they are fed with ~50000-60000 datapoints. What can I do to increase this speed?
I thought about reducing dimensionality, but after some tinkering it seemed that this reducing process cost more processing time than it actually saved.
Some other things I could try is to split the datapoints into smaller clustering groups (e.g. divided by male and female to get 2 half-as-big cluster groups), combining/averaging some or all individual items in a cluster to save datapoints or by just allowing e.g. 10 datapoints per cluster (again to save datapoints).
Since I get this data incrementally, I either have to repeat the whole clustering process (seems to be the case with DBSCAN, HDBSCAN and Chinese Whispers) which means this process needs to go as fast as possible since re-runs are costly, or by having an incremental clustering process that can work efficiently on huge datasets when added and run incrementally (so one incremental step doesn't take as long as a whole reclustering, otherwise this would be pointless), but I haven't found a algorithm yet that fulfills my requirements.
Can you lead me into the right direction? Also, it would be great if there would already exist a high performance algorithm available with a Python API, since I'm not great at highly optimizing algorithms.


texts:
I have a dataset with the following structure:

full_name,nickname,match
 Christian Douglas,Chris,1,
 Jhon Stevens,Charlie,0,
 David Jr Simpson,Junior,1
 Anastasia Williams,Stacie,1
 Lara Williams,Ana,0
 John Williams,Willy,1

where each predictor row is a pair full name, nickname, and the target variable, match, which is 1 when the nickname corresponds to the person with that name and 0 otherwise. As you can see, the way the nickname is obtained from the full name doesn't follow a particular pattern.

I want to train an ML algorithm that, given the pair full name, nickname, predict the probability of match.

My baseline is just trying to see the number of carachters that match, and features like that. However, I am thinking about an NLP approach using deep learning. My question is wether there are neural network architectures that are specific of this problem.


texts: Google prediction API: What training/prediction methods Google Prediction API employs?

texts:
The details of the Google Prediction API are on this page, but I am not able to find any details about the prediction algorithms running behind the API.

So far I have gathered that they let you provide your preprocessing steps in PMML format.


texts: Topic modelling with many synonyms - how to extract 'latent themes'

texts:
Here's my corpus
{
    0: "dogs are nice",       # canines are friendly
    1: "mutts are kind",      # canines are friendly
    2: "pooches are lovely",  # canines are friendly
    ...,
    3: "cats are mean",         # felines are unfriendly
    4: "moggies are nasty",     # felines are unfriendly
    5: "pussycats are unkind",  # felines are unfriendly
}

As a human, the general topics I get from these documents are that:

canines are friendly (0, 1, 2)
felines are not friendly (3, 4, 5)

But how can a machine find the same conclusion?
If I were to do a Latent Dirichlet Allocation approach, I feel like it would struggle to find topics because the synonyms are 'diluting' the underlying meaning. For example:

"dogs", "pooches", and "mutts" could all fall under "canines"
"nice", "kind", and "lovely" could all fall under "friendly personality trait"

Is there a way where I can use an already-trained set of latent vectors (e.g., Google News-vectors-negative300.bin.gz) to represent each document in these broader entities, and then find the topics? (i.e., instead of using the 'raw' words)
Does that even make sense?

EDIT: Come to think of it, I think my question essentially boils down to: is it possible to replace/redefine a set of similar-meaning words with a single all-encompassing word?


texts: How can I fix this error : Python Error ""ValueError: could not broadcast input array from shape (4) into shape (1000)""

texts:
I have the following python code:

from keras.applications import InceptionResNetV2
conv_base=InceptionResNetV2(weights='imagenet',
                            include_top=False,
                            input_shape=(150,150,3))
#conv_base.summary ()
# Extract features
import os, shutil
from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(rescale=1./255)
batch_size = 16

def extract_features(directory, sample_count):
    features = np.zeros(shape=(sample_count, 3, 3, 1536))  # Must be equal to the output of the convolutional base
    labels = np.zeros(shape=(sample_count))
    # Preprocess data
    generator = datagen.flow_from_directory(directory,
                                            target_size=(150,150,),
                                            batch_size = batch_size,
                                            shuffle=False,
                                            class_mode='categorical')
    # Pass data through convolutional base
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base.predict(inputs_batch)
        features[i * batch_size: (i + 1) * batch_size] = features_batch
        labels[i * batch_size: (i + 1) * batch_size] = labels_batch
        i += 1
        if i * batch_size >= sample_count:
            break
    return features, labels

train_features, train_labels = extract_features(train_dir, 384)  # Agree with our small dataset size
validation_features, validation_labels = extract_features(validation_dir, 80)
test_features, test_labels = extract_features(test_dir, 86)

I get the following error:

  File "<ipython-input-31-6bdd0a78c5f7>", line 26, in <module>
    train_features, train_labels = extract_features(train_dir, 384)

  File "<ipython-input-31-6bdd0a78c5f7>", line 20, in extract_features
    labels[i * batch_size: (i + 1) * batch_size] = labels_batch

ValueError: could not broadcast input array from the shape (16,4) into shape (16)


texts: Time series binary classificaiton with labelling issues

texts:
My situation is quite complicated so I will give a similar example from a simpler domain. Suppose we want to try to predict WHEN a mobile game users will make a purchase if given a sale. Almost every user is always instantaneously a non-purchaser because everybody is constantly not buying anything. Some people buy something to instantaneously become a purchaser but go back to the standard state of not purchasing. We have a time-stamp for that purchase. If we could understand the user behavior well enough, we could predict when a user will purchase before they do so. If this is the case then users with actions and game states very similar to the purchasers but who do not actually purchase would do so if given a sale. So the question is how do we turn this into a machine learning question. My current plan is to use binary classification by labeling the YES cases as the purchasers since those who purchase at regular price would have purchased if given a sale. I build my features based on actions in recent look back windows, like how many actions of a type in the last day. For all users who never purchase I choose random time-stamps, build my features and use them as the NO case. Since I can choose as many NO time-stamps as I want I have been doing 100 times my YES case. Then I can use a classifier, I like tree ensemble methods but I do not think the classifier will really matter here.

The problem is that this is not working. There are two things worth noting which I think are at the core of the issue I am having. First, we are trying to predict the "when" not just the "who". Users who play a lot or who have purchased before are much more likely to purchase in general so it is easy to make a time independent classifier and predict who is likely to purchase. This means that the features which are useful for the easy time-independant problem can sort of "contaminate" the feature set. I have taken out a number of these features and seen some improvement. The second issue has to do with labelling. We have many YES cases for the purchasers but how does one define a NO case. I described what I am doing above but I am not sure that it is really a best method. Many of the users I defined as a NO might have effectively been a YES since if they were given the sale they would have bought.

Also, I should point out two situations which this is not. This is not one class binary classification which generally uses the large class in the imbalanced case reducing the problem to unsupervised anomaly/outlier detection. This is also not the semi-supervised situation where I have both classes labelled for a subset of the data. I really only have some of the YES cases for sure and none of the NO case. Any thoughts welcome.


texts:
I have been given different error rates for k=1,2,3.

How to calculate the average value of error,and determine whether it is a good error rate or not?


texts: I am working on a project, and I've been trying to come up with a custom loss/score function. Would really appreciate some help

texts:
So, the loss function needs to be a function of (say) two variables A and B. It needs to be continous and differentiable in terms of A. Basically, I have a set of precomputed values (B), and during training, after every forward pass, I get a value A. A can vary from 0 to 1. And B can vary from 0 to (say) a bit large value, 10.

Now, this is how I want the loss function to work :

A - high, B - high
=> Ideal situation. Loss should be a small value i.e. No change in A required.

A - high, B - low
=> Loss should be such that it moves towards decreasing the value of A.

A - low, B - high
=> Loss should be such that it moves towards increasing the value of A.

A - low, B - low
=> Ideal situation. Loss should be a small value i.e. No change in A required.


texts:
Take the given values: -2.998, 8.0, 2.5.

My normalisation function takes a sum of the absolute values (13.498), let's call that x.

Next, each absolute value is divided by x (essentially calculating the mean distribution) - here's the result: 0.222, 0.593, 0.185.

So our result values are spread between 0-1, (it's what I need), and works great. But what's the precise terminology for this type of normalisation? I want to be able to document and comment my code accurately.


texts: How to calculate ideal Decision Tree depth without overfitting?

texts: What would be a good way to go around finding the best depth for a DecisionTree (in SKLearn)? How can I tell if I've gone too deep and am overfitting? I know I can find the best parameters with f.e. GridSearchCV, but the best score might not mean I get the best classifier as I may overfit the tree to the data.


texts:
My task is to estimate a person's age based on a face image of that person.
To that end I'm using a CNN and at first stage I was based on the following article: DeepExpectation which uses a VGG16 architecture to predict a person apparent age (the age that other people would vote). I'm using ResNet 50 architecture (and I'm using this implementation of it: ResNet Tensorflow in tensorflow). The dataset I use for learning is taken from the same article above and can be downloaded from here: WIKI-IMDB dataset. It is composed of 523,051 face images with tagged ages but I found that most of them are garbage (doesn't have real age as the label or has more than one face in it or no face at all). after filtering this dataset (by throwing images that have more than one face or no face at all or the label that represents the age doesn't make sense) I'm left with approximately 150k images in my dataset. in the process of making this dataset I centered all faces in the image and cropped it to be 224*224*3. To increase the size of my dataset I flipped horizontally each image in my dataset to get a total of 300k images in my dataset. I then split the dataset to 240k train, 30k validation and 30k test images.

I'm loading a pre-trained model (trained on ImageNet) and I hoped to get the results they got in the article above (something close to 3.2 years with MAE as the evaluation metric). I also should mention that I'm using cross-entropy as the loss and an l2 loss for regularization and I have 101 classes as the logits (ages 0-100). The batch size I'm using is 64 (The maximum my GPU can handle with). I tried several combinations of hyperparameters but until now the best MAE I got on the validation set was 5.8 years.

Examples of loss graphs I got and their hyperparameters:
weight decay = 5e-4,
momentum optimizer with momentum=0.9,
first learning rate = 0.01 and reducing it by a factor of 10 each 2 epochs until reaching to 1e-6,
the above LR is the LR assinged to the last FC layer. for the layer before it I used 0.5 of that LR. for the middle layers I used 1e-2 of that LR and for the first layer I used 1e-3 of that LR.
I got the following graphs: (orange = train , blue = validation)
  loss:
  LR:
  MAE:

example #2:
same hyperparameters as in the example above only different ratios between first layers and last FC layer. in this example I used 0.5 of the LR in the last FC layer for the layer before it and 0.1 of the LR in the last FC layer for the other layers.
I got the following graphs: (orange = train , blue = validation)
  loss:
  LR:
  MAE:

As you can see from the last 2 examples, I'm reaching to ~6 pretty fast but it seems as the optimization on the train loss gets stuck and also that the validation loss doesn't approach the train loss from some point.
Do you have any suggestions?

Also, I'm having some thoughts on whether it is good to apply some of the following:

Initalize weights from the pre-trained network until some layer
Initialize all weights from the pre-trained network beside the batch-norm
Freeze some of the layers after loading them and train only the last layers
Apply different LR to different layers (as I did in the examples above)
Should I use Momentum optimizer or maybe Adam Optimizer
If I'm using Momentum optimizer, what should be the learning rate schedule (learning rate decay)?
If I'm using Adam optimizer, does it make sense to use different learning rates to different layers?
What should be the weight decay hyperparameter?

Any help would be much appreciated.

I didn't know which stackexchange site is more appropriate for this question so I also posted this question on CrossValidated: Difficulty in choosing Hyperparameters for my CNN


texts: Why are results without Transfer Learning better than with Transfer Learning?

texts:
I developed a neural network for license plate recognition and used the EfficientNet architecture (https://keras.io/api/applications/efficientnet/#efficientnetb0-function) with and without pretrained weights on ImageNet and with and without data augmentation. I only had 10.000 training images and 3.000 validation images. That was the reason I applied Transfer learning and image augmentation (AdditiveGaussianNoise).
I created this model:
efnB0_model = efn.EfficientNetB0(include_top=False, weights="imagenet", input_shape=(224, 224, 3))
efnB0_model.trainable = False

def create_model(input_shape = (224, 224, 3)):
    input_img = Input(shape=input_shape)
    model = efnB0_model (input_img)
    model = GlobalAveragePooling2D(name='avg_pool')(model)
    model = Dropout(0.2)(model)
    backbone = model

    branches = []
    for i in range(7):
            branches.append(backbone)
            branches[i] = Dense(360, name="branch_"+str(i)+"_Dense_360")(branches[i])
            branches[i] = BatchNormalization()(branches[i])
            branches[i] = Activation("relu") (branches[i])
            branches[i] = Dropout(0.2)(branches[i])
            branches[i] = Dense(35, activation = "softmax", name="branch_"+str(i)+"_output")(branches[i])

    output = Concatenate(axis=1)(branches)
    output = Reshape((7, 35))(output)
    model = Model(input_img, output)

    return model

I compiled the model:
opt = keras.optimizers.Adam(learning_rate=0.0001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=["accuracy"])

And used this code to fit it:
hist = model.fit(
    x=training_generator, epochs=10, verbose=1, callbacks=None,
    validation_data=validation_generator, steps_per_epoch=num_train_samples // 16,
    validation_steps=num_val_samples // 16,
    max_queue_size=10, workers=6, use_multiprocessing=True)

My hypotheses were:
H1: The EfficientNet architecture is applicable to license plate recognition.
H2: Transfer learning will improve accuracy in license plate recognition (compared to the situation without Transfer Learning).
H3: Image augmentation will improve accuracy in license plate recognition (compared to the situation without it).
H4: Transfer Learning combined with Image augmentation will bring the best results.
I now got this results:

So, H1 seems to be correct. But H2, H3 and H4 seem to wrong.
I was thinking about it and got an explanation for H3 and H4, which seem to be logical for me. That is, that image augmentation is too heavy and deteriorates the quality of images by a degree which makes it very hard for the network to recognize the characters.
1. Is this a suitable explanation and are there other ones additionally?
It seems to be the case, that image augmentation was too strong. So, first question is solved.
Regarding H2 I am little confued to be honest. The network seems to overfit but stagnates completely regarding validation accuracy. So, the conclusion that the Imagenet weights are not applicable seems not logical to me because the network learnt something for the training data. I also excluded the possibility that the data volume is to small since we had that good recognition rates without using Transfer learning or image augmentation...
2. Is there any logical explanation for this?


texts: Weka alternative - visual data exploration with a very large dataset

texts:
I love Weka for its visual data exploration features (it's great at quickly giving you a mile-high aerial view of the distributions of all your variables).

I have a 0.5gb dataset, and Weka is unable to load it (on a 64-bit Windows PC with 16gb of RAM).

For the Weka fans out there, what's the closest 'visual data exploration' alternative for very large datasets?

(FWIW - I've done all the various things that are recommended when Weka fails at loading a very large dataset, e.g., increasing the Java heap size, etc.)

Update: I tried RapidMiner Basic (it's open source and free), and that worked.


texts: What does the collate function in pytorch (geometric)?

texts:
I am creating a message passing neural network and have some issues with the dataset creation. In pytorch (geometric) it is recommended to create a dataset with the following class. I wonder what is the meaning of the collate function that is called at the end of the process method? In what cases should I use my own collate function? My graphs have mostly different sizes.

import torch
from torch_geometric.data import InMemoryDataset

class MyOwnDataset(InMemoryDataset):
    def __init__(self, root, transform=None, pre_transform=None):
        super(MyOwnDataset, self).__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self):
        return ['some_file_1', 'some_file_2', ...]

    @property
    def processed_file_names(self):
        return ['data.pt']

    def download(self):
        # Download to `self.raw_dir`.

    def process(self):
        # Read data into huge `Data` list.
        data_list = [...]

        if self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(data) for data in data_list]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])

The error I get is:

  RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 4422 and 4032 in dimension 1 at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/generic/THCTensorMath.cu:71


texts: What algorithm i use for a Wikipedia bot?

texts:
i want to create a bot for Wikipedia which fight vandalism.
It's like spam detection but people can remove text or add text. I want to use machine learning to do this. I don't know which machine learning algorithm should i use.


texts: Is there a way to compute micro-average for NDCG

texts:
I know how both micro and macro averages computed and there is a great discussion of the topic: Micro Average vs Macro average Performance in a Multiclass classification setting
The macro-average for NDCG is straightforward, but can we have something like micro-average for it?


texts:
In the field of machine learning, I'm wondering about the interest of applying feature selection techniques.

I mean, I often read articles or lectures speaking about how to reduce the number of feature (dimensionality reduction, PCA), how to select the best features (feature selection etc).

I'm not sure of the main purpose of this:

Does feature reduction techniques always improve accuracy of the learned model?
Or is it just a computational cost purpose?

I would like to understand when it is necessary to reduce the number of features and when it is not, in order to improve interpretability or accuracy.
Thanks!


texts: How to methodologically show that a given 'time-series/sequential' data is not really sequential?

texts:
I have an apparent time-series/sequencial (supervised: multi-class classification problem) dataset with each data-point time-stamped. However, some domain intuition tells me that the data is static and no need to treat them as time-series data. I have also verified that a non-timeseries method fits (even with rigorous cross-validation and other good practices) the data well. However, I want to prove this in more rigorous/methodological way, preferably in python but R is also fine if no alternative. Is there any method/test that validate my hypothesis?

Thanks.


texts: Good "frequent sequence mining" packages in Python?

texts: Has anyone used (and liked) any good "frequent sequence mining" packages in Python other than the FPM in MLLib? I am looking for a stable package, preferable stilled maintained by people. Thank you!


texts: How to model a non linear distribution using logistic regression

texts:
To put us on the same page the goal for a logistic regression is to determine a vector of real numbers which will serve as coefficients producing a linear combination of the feature vector values (plus a constant).
That said, I'm now going to present a learning question and a simplified data set, and a naive approach to engineering a feature for the training. I'm then going to point out why based on the algorithm, I think this feature engineering choice is a bad fit for this algorithm and what approach I think is better suited.
I'm brand new to ML/Stats and I'm hoping that someone with more experience/insight can validate where I am on track and point out where I am off track, so here goes:
I have a a set of made phone calls, labeled as 'picked up' and 'did not pick up', or {1,0}, respectively.
I have a timestamp for when the call was made. Let's assume all callees are in the same timezone to simplify everything.
Learning Question: how does the hour of day of the call affect the pick up rate for the call P( y = 1 | x) where x = hour of day.
Here's the naive approach to engineering the timestamp feature:
Transform the timestamp into the hour of the day [1,24], and train the model with it.
Here's where I think this is a very bad way to engineer the hour of day feature:
Since all the algorithm will do is find the optimal coefficient for hour of day, it has no choice but to see call rate (more or less, depending on feature cardinality) as a linear function of hour of day. Meaning that the coefficient can be set to reward either a higher value or a lower value, with the predictive assumption being that the probability for a pick up will be directly or inversely proportional to the integer value of day.
Thus this feature engineering approach makes it so that model cannot speak at all to a probability distribution that is not linear, like say when calls made at hour 8 have a relatively high pick up rate, but calls made at 9 do not, but those made at 12 do, and so forth...
I don't think this changes when there are more features, because at the end of the day the algorithm is limited to defining a scalar to represent the relationship between any given feature and the target label.
QUESTION: Does this makes sense? Is this analysis correct or not or something in between?
A better approach (I'd like to say even a good approach, please tell me what you think) is to one hot encode the hour of day such that hour of day becomes an individual feature with its own fine tuned coefficient.
QUESTION: Does this makes sense? Is it a better approach? A good approach or not or something in between?


texts: How do I download a Google Colab notebook?

texts: I have to download code written in a Google Colab notebook, because I need to send it via email. How do I do this?


texts: Using a model to generate training data for another model

texts:
I have a text classification problem. Lets say I need to classify text into labels A,B,C & D.

I have two different approaches and want to understand the difference:

Approach 1: Traditional approach of proving labeled training data to a single label classifier model and training it

Approach 2:

Step 1: Building 4 different models, one for each label, that predicts the likelihood of that label (Yes/No). This will use the same training data is Approach 1 split in 4 ways, one for each label, with additional annotations for "No" label.

Step 2: Using the above models to generate labeled training data with tens of thousands of rows with 99.9% accuracy (discard any text where the previous model has low score), and all four labels A,B,C,D

Step 3: Use the generated data to train new model to predict the label for a given text.

I'm new to NLP so would like to understand, is Approach #2 going to be better than #1 in terms of precision and recall? I assume it will be, but what doesn't make sense intuitively is that Approach 2 is using the same human labeled data as input as Approach 1 (with exception of some additional data in Step 1 for the "No" Labels. Is the accuracy of the models going to be much different?


texts: Detect constant (zero-slope) sections in a noisy step function

texts:
I have data that has the shape of a step function, which is obstructed by noise. I would like to identify the sections with constant slope in it.

The noise does not necessarily have 0 mean, and the steps are not always abrupt. Also, the points on the x-axis are not equidistantly spaced.

I was thinking about trying one-sided differentiators (like here) but not sure if they would work with non-equidistantly spaced data.

I feel like some sort of a global approach based on optimization would be an elegant way to solve that but so far I couldn't come up with one.


texts: Sort a data frame column based on another sorted column value in R

texts:
I have a data frame that is sorted based on one column (numeric column) to assign the rank. If this column value is zero then arrange the data frame based on another character column for those rows which have zero as a value in a numeric column.

But to give rank I have to consider var2 that is the reason I sorted based on var2, if there is any identical values in var2 for those rows I have to consider var3 to give rank. Please see the data frame 2 and 3 rows, var2 values are identical in that case I have to consider var3 to give rank. In case var2 is zero I have to sort the var1 column (character column) in alphabetical order and give rank. If var2 is NA no rank. Please refer the data frame given below.

Below, the data frame is sorted based on var2 column descending order, but var2 contains zero also if var2 is zero I have to sort the data frame based on var1 for the rows which are having zero in var2. I need sort by var1 for those rows which are having var2 as zero and followed by NA in alphabetical order of var1.

    example:
    #      var1    var2    var3    rank
    # 1     c      556      45       1
    # 2     a      345      35       3
    # 3     f      345      64       2
    # 4     b      134      87       4
    # 5     z       0       34       5
    # 6     d       0       32       6
    # 7     c       0       12       7
    # 8     a       0       23       8
    # 9     e      NA
    # 10    b      NA

below is my code
df <- data.frame(var1=c("c","a","f","b","z","d", "c","a", "e", "b", "ad", "gf", "kg", "ts", "mp"), var2=c(134, NA,345, 200, 556,NA, 345, 200, 150, 0, 25,10,0,150,0), var3=c(65,'',45,34,68,'',73,12,35,23,34,56,56,78,123))

# To break the tie between var3 and var2
orderdf <- df[order(df$var2, df$var1, decreasing = TRUE), ]

#assigning rank
rankdf <- orderdf %>% mutate(rank = ifelse(is.na(var2),'', seq(1:nrow(orderdf))))

The expected output is sort the var1 in alphabetical order if var2 value is zero (for those rows with var2 value is zero).

    expected output:
    #      var1    var2    var3    rank
    # 1     c      556      45       1
    # 2     a      345      35       3
    # 3     f      345      64       2
    # 4     b      134      87       4
    # 5     a       0       34       5
    # 6     c       0       32       6
    # 7     d       0       12       7
    # 8     z       0       23       8
    # 9     b      NA
    # 10    e      NA


texts: is it possible to train a model which detects and draws bounding boxes for objects on a table, when I use a dataset where objects on a table are labeled with boundingboxes?


texts:
I want to clarify point in random forest literature. In random forest we select random features from a subset of data, i.e it is subset of features from subset of data for a single tree. But when we test the data, let's take a test sample 'v' from the test data. The data has say '5' features.

Tree one was constructed using features 3 and 2.
Tree two was constructed using features 2,3, and 5.
Tree three was constructed using features 1,2 and 3.
Tree four was constructed using features 1,4,and 5.

at leaf node we will have probability showing to which class does the 'v' point belongs.

My question is : Will each forest choose same features from the test data sample 'v' as it used when the tree was constructed. ?
for example will tree one will use features 3 and 2. and tree two use feature 2, 3 and 5. respectively


texts:
From Hadoop The Definitive Guide

  The whole process is illustrated in Figure 7-1. At the highest level,
  there are five independent entities:

  • The client, which submits the MapReduce job.

  • The YARN resource manager, which coordinates the allocation of
  compute re‐ sources on the cluster.

  • The YARN node managers, which launch and monitor the compute
  containers on machines in the cluster.

  • The MapReduce application master, which coordinates the tasks
  running the Map‐ Reduce job. The application master and the MapReduce
  tasks run in containers that are scheduled by the resource manager and
  managed by the node managers.

What is the MapReduce application master?

In a MapReduce program written in Java, we need three things: a map function, a reduce function, and some code with main() function to run the job. Is  the MapReduce application master the code with main() function to run a map reduce job?

Thanks


texts: Do I need to convert booleans to ints to enter them in a machine learning algorithm?

texts:
My dataset contains a lot of columns with booleans do I really need to change them so I can insert them into the algorithm?

I'm gonna use KNN right now but will test other algorithms later so I'm trying to ready up my dataset


texts: Distinguising features of linear vs, non-linear machine learning models (algorithms)

texts: What are some examples of linear and non-linear machine learning models (algorithms) for purposes of comparison between the two categories? Which are the parameters (or scalars in a linear algebraic sense) and which are the predictors/factors (or vectors in a linear algebraic sense again)?


texts: Is plant classes or any crop is present in Imagenet datasets?

texts: I am working on a project, which is based on plants. I only want to know that whether Imagenet datasets include tree, crop classes or not.


texts: Should I normalize the target data the same way as the input data?

texts:
This is the way I'm currently transforming my input data

x_scale = preprocessing.MinMaxScaler()
training_x = x_scale.fit_transform(training_x)
testing_x = x_scale.transform(testing_x)
validation_x = x_scale.transform(validation_x)

Should I be processing the target data the same way?

y_scale = preprocessing.MinMaxScaler()
training_y = y_scale.fit_transform(training_y)
testing_y = y_scale.transform(testing_y)
validation_y = y_scale.transform(validation_y)

If my activation is linear and error is mse, is this way of normalization correct?


texts: What exactly is the input of decoder in autoencoder setup

texts:
I am reviewing various autoencoder setups for MNIST reconstruction, Seq2Seq translation and others.
My naive understanding of data flow is as follows:

Input -> [Encoder] -> Hidden Representation -> [Decoder] -> Output.

However, in case of Seq2Seq translation task similar to Sutskever et. al. decoder input is combined from hidden state + input sequence.

I wonder how is the input of decoder dependent on the target task? Why do we need to put input sequence besides the hidden state? Any high-level explanation is appreciated.


texts: How to load large csv file from web in python

texts: My code is in jupiter notebook and my data set(400-800mb) is present on google drive/dropbox. My task is to load the csv file from another server into my jupiter notebook. How can I achieve this? Should I move my csv file to some other server?


texts: Python - Create many dummy variables from one text variable?

texts:
I'm trying to create dummy variables for a variable that has text data in rows.

Data in 1st row is:
{"Wireless Internet","Air conditioning",Kitchen,Heating,"Family/kid friendly",Essentials,"Hair dryer",Iron,"translation missing: en.hosting_amenity_50"}

and Data in 2nd row is:
{TV,"Cable TV",Internet,"Wireless Internet",Kitchen,"Indoor fireplace","Buzzer/wireless intercom",Heating,Washer,Dryer,"Smoke detector","Carbon monoxide detector","First aid kit","Fire extinguisher",Essentials} and many more.

What I now want to do is, to create dummy variables out of that variable. For example from the above data:
one variable named Wireless Internet with 0 ans 1 in rows &
another variable named Cable TV with 0 and 1 in rows &
another variable named Kitchen with 0 and 1 in rows and so on.

sklearn for python has OneHotEncoder class which creates dummy variable named everything in a row considering all rows with unique values. That is not what I want to do here. I first have to split text in all rows and create dummy variables for them. How do I do that?

Expected results are, multiple columns like
Wireless Internet    Cable TV      Kitchen
1                     0             1
0                     1             1
1                     0             1

link to data(column named amenities) - https://www.kaggle.com/stevezhenghp/airbnb-price-prediction


texts: Can inferencing come from incomplete rule sets?

texts:
I have some data for medical diagnosis, consisting of some rules about relationship of diseases and their symptoms, for example disease D1 frequently has symptom S1 or disease D2 rarely has symptom S1. Given some symptoms, I want to somehow calculate most probable diseases based on this data.

I thought about using a naive bayes model, but using that needs some assumptions about disease prevalence or probability of some symptoms given a disease that are not present in my rules set. Is there an standard way of doing this kind of analysis? For example are fuzzy systems used in this situation?

NOTE: I think that in the cases where relation between a disease and a symptom is not explicitly stated as a rule it is an indication of either independence or a very low probability of the symptom given that disease.


texts: How to build a model where multiple data points contribute to a result

texts:
I’m trying to figure out how to massage data and model the following scenario:

Customers at a restaurant rate the quality of the service between 1-10.

I have data on individual interactions between the servers and customers. Say - length of interaction, type of interaction (refilling beverage, ordering, cleaning, etc).

Hypothesis here is each interaction contributes to the final score. I want to build a model that tells me given an interaction, how does it move the score.

My intuition is if I arranged the data as individual interactions, with output of final score, that’ll give me what I want. Is that true?


texts: Can continuous variables decrease classification model accuracy?

texts: I have been playing around with titanic dataset. Here Fare column is a continous variable. I've read people stating that in a classification model it's best to have categorical variables than continous features. So I was wondering if I convert the age into categorical bins will it improve my model accuracy?


texts:
For what I read the 5x2cv t test is

"a procedure for comparing the performance of two models (classifiers or regressors) that was proposed by Dietterich to address shortcomings in other methods such as the resampled paired t test and the k-fold cross-validated paired t test"

.

I am currently making some experiments with an unbalanced data set, which I balanced using SCUT and trained a set of different classifiers. The problem is a multiclass with three different classes to choose. I am applying a Multilayer Perceptron, a Decision Tree and a Random Forest and the results, after a 10-fold cross validation, are the following:

Multilayer Perceptron: 0.95 acc

Decision Tree: 0.93 acc

Random Forest: 0.935 acc

When I apply the 5x2cv t test I got the following results:

MLP and DT:

t-statistic 4.75

p-value 0.005

So, if I assume that for the tests I will have a value of alpha of 0.05 for rejecting the null hypothesis, which is that both algorithms perform well with the same database, then from the p-value I got I could reject the null hypothesis. This would mean that both models do not perform equally well, so it would be better to use MLP over the DT because of its higher accuracy.

When I do the same with the MLP and the RF I got the following results:

t-statistic: 2.46

p-value: 0.055

Here, I suppose that I can conclude that using MLP or RF for the current dataset is almost the same, because I failed to reject the null hypothesis. The question that I have here is if should I choose the RF even though the accuracy was lower?

The comparison with the DT and RF has the following values:

t-statistic: -2.49

p-value: 0.054

So I can reject the null hypothesis and said that there is difference between the use of the DT over the RF.

Are my conclusions correct?

Thanks


texts:
I have a dataset of drone captured images (non-geotaggged) which I would like to stitch to form a single image. I wanted to know what type of a neural network can I use to associate multiple images to a single image? & how do I feed multiple images as X to fit a single image (Y)
Thanks.


texts:
I have a binary car sound classifier. I have a feature set that is extracted from audio with size of 48. I have a model(multi layer neural network) that has around %90 accuracy on test and validation sets. (without normalization or Standardization)

I see that the feature values are mostly around [-10, +10] But there are certain features with a mean of 4000. Seeing unproportional values within features I though some feature scaling might improve things. So using scikit-learn tools I tried:

   - Simply removing the means from features
   - Normalizer
   - Min max scaler
   - Robust Scaler

And all these above ended up dropping my accuracy to ~ %50!  (%100 recall, %50 precision)

So how is this possible? And what is the correct way to normalize my data?


texts: What does sklearn's pairwise_distances with metric='correlation' do?

texts:
I've put different values into this function and observed the output. But I can't find a predictable pattern in what is being outputed.

Then I tried digging through the function itself, but its confusing because it can do a number of different calculations.

Docs

According to the docs:

Compute the distance matrix from a vector array X and optional Y.

I see it returns a matrix of height and width equal to the number of nested lists inputted, implying that it is comparing each one.

But otherwise I'm having a tough time understanding what its doing and where the values are coming from.

Examples I've tried:

pairwise_distances([[1]], metric='correlation')
>>> array([[0.]])

pairwise_distances([[1], [1]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

# returns same as last input although input values differ
pairwise_distances([[1], [2]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])

# returns same as last input although input values differ
# I incorrectly expected more distance because input values differ more
pairwise_distances([[1,2], [1,3]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>       [2.22044605e-16, 0.00000000e+00]])

Computing correlation distance with Scipy

I don't understand where the sklearn 2.22044605e-16 value is coming from if scipy returns 0.0 for the same inputs.

# Scipy
import scipy
scipy.spatial.distance.correlation([1,2], [1,2])
>>> 0.0

# Sklearn
pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])

I'm not looking for a high level explanation but an example of how the numbers are calculated.


texts: Can I download Twitter data via web scraping for research?

texts:
I want to do a sentiment analysis using twitter data. Was thinking about hardcoding a cURL script to download data, from a Google Cloud service (I'll run the data on a neural network on the server, to label each tweet), but I have this question:

Am I allowed to do this? I know twitter sells the data, so I am not sure if I can get in trouble for downloading it directly (I have to disclose the data gathering methodology on the paper).


texts: Dataset: language audio clips and country labels

texts:
I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from.

Does such a dataset exist? Or is there an easy way to hack one together?


texts: How to deal with a feature that has lot of categorical values?

texts:
I know this question has been asked before and I have tried a few things but those things are not working as expected for my usecase.

I have a 500 length feature vector. One of these features is a categorical value pincode. For our dataset, the pincode can take more than 20,000 unique values. So we can't used one hot encoding because it will blow up our feature space.

I have also tried Binary Encoding which assigns a unique integer to every unique categorical value and then converts it into Binary. It then treats every bit as a column. This way the dimension of the pincode feature vector had reduced from 20,000 to around 20. But I didn't find any improvement in our evaluation metrics which is AUC (Area under Curve). Intuitively also it didn't felt right thing to do.

I also tried to apply PCA on the pincode to reduce the dimensions. I tried to reduce the dimensions from 20,000 to 100. Upon applying PCA, the model performed worse. Also, I read somewhere that PCA doesn't work well on categorical values. It's better for continous values.

So what do we do with this feature? We don't want to throw it away as we think that it might be an important feature. But we want to reduce it dimension and then use.

Apologies if this is a basis question. I am new to this field and trying various things out.

P.S - We are using xgboost for training our model.


texts:
I am not sure how to formulate this problem clearly into a machine learning task yet. So hope you guys can chime in and give me some help.

Problem : To predict whether someone will pick up their phone during office hours in week n+2 by looking at customer's behaviour in week  n.

Data    : I have calling records for about 3 months, which is aggregated on customer level. The various attributes include, num of calls, duration of calls, time of calls, amount of data traffic. But of course, these main attributes are further split into at about 20 attributes.

Current Approach (Very Manual) : I look at data at week n+2 and get the group of guys who picked up the phone during office hours (duration of calls > 5s and time of call). This is the target group, T.
I look at data at week n and manually try all possible combinations of the attributes to get as close to T as possible. But trying manually seems tiring after some time. The baseline is of course using the same conditions as at week n+2. But the whole idea will be to increase this number.

Question : Is there any way I can transform this dataset so that I would be able to do accomplish it as a machine learning task ?


texts: Big difference in randomTree accuracy with train and test sets

texts:
I've created a model using randomForest for the following dataset: https://archive.ics.uci.edu/ml/datasets/Contraceptive+Method+Choice

The thing i'm questioning is that the results of the model when used on my training and testing sets are vastly different.

library(randomForest)
library(caret)
df <- read.csv('cmc.csv')

Changing values to factors

df$Wife.s.education <- as.factor(df$Wife.s.education)
df$Husband.s.education <- as.factor(df$Husband.s.education)
df$Wife.s.religion <- as.factor(df$Wife.s.religion)
df$Wife.s.now.working. <- as.factor(df$Wife.s.now.working.)
df$Husband.s.occupation <- as.factor(df$Husband.s.occupation)
df$Standard.of.living.index <- as.factor(df$Standard.of.living.index)
df$Media.exposure <- as.factor(df$Media.exposure)
#add string representation for readiblilty
df[df$Contraceptive.method.used == 1,]$Contraceptive.method.used <- "No-use"
df[df$Contraceptive.method.used == 2,]$Contraceptive.method.used <- "Long-term"
df[df$Contraceptive.method.used == 3,]$Contraceptive.method.used <- "Short-term"
df$Contraceptive.method.used <- as.factor(df$Contraceptive.method.used)

Splitting data:

set.seed(47)
ind <- sample(2, nrow(df), replace = TRUE, prob = c(0.7,0.3))
inTrain <- createDataPartition(y=df$Contraceptive.method.used, p=.7, list = FALSE)
training = df[ind==1,]
testing = df[ind==2,]

Creating model:

model <- randomForest(Contraceptive.method.used ~., data = training, proximity=TRUE)

#Prediction & Confusion Matrix - training data
p1 <- predict(model, training)
confusionMatrix(p1, training$Contraceptive.method.used)

Confusion Matrix Results (training):

          Reference
Prediction   Long-term No-use Short-term
  Long-term        208      9         18
  No-use             4    408          5
  Short-term        16     14        338

Overall Statistics

               Accuracy : 0.9353
                 95% CI : (0.9184, 0.9496)
    No Information Rate : 0.4225
    P-Value [Acc > NIR] : < 2e-16

                  Kappa : 0.9002

 Mcnemar's Test P-Value : 0.09773

#Prediction & Confusion Matrix - testing data
p2 <- predict(model, testing)
confusionMatrix(p2, testing$Contraceptive.method.used)

Confusion Matrix Results (testing)

    Reference
Prediction   Long-term No-use Short-term
  Long-term         42     11         23
  No-use            27    122         47
  Short-term        36     65         80

Overall Statistics

               Accuracy : 0.5386
                 95% CI : (0.4915, 0.5853)
    No Information Rate : 0.4371
    P-Value [Acc > NIR] : 8.972e-06

                  Kappa : 0.2788

 Mcnemar's Test P-Value : 0.005869

As we can see theres is a massive change in the two results, if i print my model I get the following results:

     Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 3

        OOB estimate of  error rate: 46.96%
Confusion matrix:
           Long-term No-use Short-term class.error
Long-term         76     63         89   0.6666667
No-use            45    282        104   0.3457077
Short-term        72    106        183   0.4930748

This makes me belief that the test results are correct however I'm not sure why there is such a big difference when used on the training set, is this due to overfitting? If so how can this be handled?

Any guidance would be awesome.


texts: How can I prepare my data from multiple time series sources for time series regression?

texts:
I have multiple sensors providing time series sources with slightly different time stamps and different sampling rates. For example I have a breathing rate feature from 1 sensor at 1 Hz, EKG from another at 100 Hz and temperature for another at 10Hz.

If I were to use this data with a 1D CNN or LSTM, how would I prepare it? I tried resampling all the sources to 10Hz and merging them by closest timestamp but that didn’t work out too well. What are common methods of tackling this problem?


texts: Isolation forest sklearn contamination param

texts:
I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm.
I'm developing in Python, more in detail using sklearn.

I found out a lot of examples on this, but what is not very clear, is how to set the contamination param during the instantion of IsolationForest.

Looking the documentation, contamination is

  the amount of contamination of the data set, i.e. the proportion of outliers in the data set.

Should I use some statistical techniques to identify this percentage?

Thank you


texts: How to convert an array of numbers into probability values?

texts:
I would like some help with respect to certain numerical computation. I have certain arrays which look like:
Array 1:
[0.81893085, 0.54768653, 0.14973508]

Array 2:
[0.48078357, 0.92219683, 1.02359911]

Each of the three numbers in the array represents distance of a data point from the cluster centroid in k-means algorithm. I want to convert these numbers into probabilities. The element which has a high distance should be converted into a low probability. For example, [0.81893085, 0.54768653, 0.14973508] can be converted into a probability vector like [0.13, 0.22, 0.65]. As it can be seen, the elements which have a high value in the original array have low value in the probability array (and of course the values in the probability array sum to 1).

Is there any mathematical technique that will achieve this result?

What I have tried till now is, I took the inverse of each of the values in the original array:

1/[0.81893085, 0.54768653, 0.14973508] = [1.22110431, 1.82586195, 6.67846172]

And then I input the resulting array to softmax function (softmax function converts an array of numbers to probabilities) - https://en.wikipedia.org/wiki/Softmax_function

This gives a probability vector of [0.00421394, 0.00771491, 0.98807115].
Is this a good approach? Is there any other approach?


texts: Can one still train a classifier with an unbalanced data set?

texts:
I want to train a binary Naive Bayes classifier. The problem is, is that I have an unbalanced set at my disposal, where the ration between the two classes is roughly 2:1 (250 examples from the first class and 100 from the second class).
Assuming that oversampling/undersampling of the data is not an option, would it still be possible to train the classifier with a 2:1 ration of the classes?
Or is this a really bad idea?


texts: Obtaining consistent one-hot encoding of train / production data

texts:
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination of categorical and numerical columns.

dummified_data = data.get_dummies()
train_data = dummified_data[:10000]
test_data = dummified_data[10000:12000]

Currently, I have a hand-written function that takes user-inputted data and transforms it into a format like dummy data. This doesn't seem sustainable as the number of columns/the size of my categorical variables grows.

Is there a way to dummify training data and production data consistently?


texts: Replace values before and after a certain value

texts:
I have a variable like this:
V1:
0
0
0
0
0
1
0
0
0
0
0
2
0
0
0
0
0

I want to change the 0 before the 1 to 1, and the 0 after the 2 to 2. It is supposed to look like this:
V1:
1
1
1
1
1
1
1
0
0
0
0
0
0
2
2
2
2
2
2

Can anyone help me? I'm using R
Thank you :)


texts: Probabilities of a Poisson distribution not making sense

texts:
I am trying to find probabilities of orders a restaurant gets on Sunday's. For last 6 months average orders are 1000 without any big anomalies like 700 or 1300. This is a case of poisson distribution & I used scipy library in python and plotted the probabilities.
The graph shows probabilities that of getting orders greater than values on x-axis, e.g. probability of orders>800 is 1, >950 is close to 0.95, >1000 is close to 0.5 & so on.
I am not sure if these probabilities are  correct esp. for high order counts, e.g. for >1050 orders prob is less than 0.1, for >1100 almost zero which I find odd because its quite likely to have 1100 or little more orders occasionally. So what are thoughts on probabilities calculated using Poisson distribution.
prob_arr=[]
orders_arr=[]

dist=poisson(1000) #mu=1000

for num_orders in range(800,1201,50):
    prob_arr.append(dist.sf(num_orders)) #survival function
    orders_arr.append(num_orders)

plt.figure(figsize=(12,4))
plt.plot(orders_arr,prob_arr,linewidth=2)
plt.xlabel("Orders Count")
plt.ylabel("Probability")
plt.title("Probabilities of Minimum Orders Count");

As suggested by @Romain Reboulleau tried truncated norm distribution and below graph shows probabilities of getting orders >than values on x-axis derived using  scipy truncnorm.

Problem is low probability of getting orders >=1100, it is 1.3% as per truncnorm vs only 0.09% with poisson.  This would make non-stats business managers laugh and I am not sure how to justify this.


texts: Definition of obstacles in new OpenAI gym environment

texts:
I study a Reinforcement Learning algorithm that navigate an agent from one initial point to another in a complex environment where other agents and obstacles exists too.
I want to make my own gym environment in OpenAI to train my model. I have read that το built your own environment requires the definition of a class and functions such as init,reset,step,render etc.

Μy difficulty is in how I will design the environment that I will train my agent. I mean the plan of the room in which my agent has to navigate.How I will make the algorithm understand that my agent has found an obstacle(lets say a static obstacle). Don't this have to be done in the beginning somehow?

I would appreciate if someone could help me!!!
Thank you in advance!!!


texts: Comparision between SSIM and MAD Image Quality Assessment Algorithms

texts:
I have been working on Most Apparent Distortion(MAD) tool to evaluate the quality of images. I have read a paper that compares SSIM, PSNR, FSIM, etc. with MAD. I am uncertain about some calculations done in the paper.

(https://arxiv.org/pdf/2005.01338.pdf)

The output of MAD gives the distance between two images between [0, infinity). How do you convert this value in the similarity of range [0,1], so that it can be compared with other metrics (ssim, ms-ssim)?


texts: Deep learning performance on classifying simple geometric figures

texts:
I'm trying to find papers or just performance data on classifying simple geometric shapes, e.g. the first six convex regular polygons. The input data is computer graphics generated, producing high contrast, clean images of convex regular polygons in various scales and orientations. One polygon per image. There is no limit, other than reasonable CPU time, on number of training images used. Testing is also done on randomly generated CG images.

Regular polygons are just an example. You can add circles, ellipses, non-regular polygons, etc. Basically any simple geometric shape, which a competent elementary school student can easily classify with 100% accuracy.


texts: Can we use Shap to interpret output changes?

texts:
Can we calculate the difference between Shapley values to interpret changes in the output? More precisely, if we get Shapley values for two different inputs, can we compare them to understand how much each feature led to the increase/decrease of the output?
Example: Let's say that I have a regression model with 10 features as input: $x=(x_{1}, x_{2}, x_{3}, ...,x_{10})$. I get two predictions: $y^{1}=70$ and $y^{2}=100$ and I want to understand how much each feature has led to such an increase (+30). Let's say that for the two inputs $x = (x_{1}, x_{2}, x_{3}, ..., x_{10})$ and $x' = ({x_{1}', x_{2}', x_{3}', ..., x_{10}'})$ I get the following Shapley values: $\phi = (7, -4, 6, 18, 7, 0, 14, -3, 17, 7)$ and $\phi' = (7, -6, 6, 20, 7, 0, 32, -3, 27, 9)$. Here I'm assuming that the expected value of all the prediction is 1 (this is why the Shapley values sum up to 69 and 99 respectively).
I calculate $\phi'-\phi = (\phi'_{1}-\phi_{1}, \phi'_{2}-\phi_{2}, ..., \phi'_{10}-\phi_{10})$ which in my example is going to be equal to: (0, -2, 0, 2, 0, 0, 18, 0, 10, 2).

Is it correct to say that the 2nd feature led to a decrease of -2, the 4th feature led to an increase of 2, the 7th feature to an increase of 18, etc.?

I believe this can be inferred somehow from the additive property of Shapley values, but I'm not finding the connection yet.


texts: Is there a difference between on-line learning, incremental learning and sequential learning?

texts:
What I mean is the following: Instead of processing all the training data at once and calculating a model, we process one data point at a time and update the model directly afterwards.

I have seen the terms "on-line (or online) learning" and "incremental learning" for this. Is there a subtle difference? Is one term used more frequently? Or does it depend on the research community?

Edit: The Bishop book (Pattern Recognition and Machine Learning) uses the terms on-line learning and sequential learning as synonyms but does not mention incremental learning.


texts: Classification or regression? Which model is more accurate if I only care about being above or under the threshold?

texts:
If I have a regression problem that can also be a classification problem by converting a continuous variable to a binary depending on a threshold, which model would be more accurate if I only care about being above or under the threshold?

Example: I have a column with house prices [20, 30, 40, 70] but I only care if the house is expensive (more than 40) or cheap (less than 40)


texts:
Why the area under the ROC Curve for a random classifier is equal to 0.5 and has diagonal shape?
For me a random classifier would have 25% of TP,TN,FP,FN and therefore it would only be a single point on the ROC Curve.


texts:
I'm new in Python and definitely, I'm sure I do some mistakes.
Here is my problem and thanks all for your help in advance.
I have 2 files (one is Hive) and the other is CSV and merging them. I have 64GB memory and I believe the CSV file I create is around 25+ GB.
My problem is when I connect remotely, I see that memory usage hits to 100%, and then I cannot even connect my workstation remotely and it needs a hard boot.
What I'm thinking is,
when I merge these 2 tables, I like to save in CSV (let's say 100,000 rows) and clean that from the memory and continues with another 100,000 rows, append to it, and so on...
I'm not sure how to do this, I found some within Google search, most likely is about to read large files, but not sure after I read (merge or during merge in my case), to write every 100K chunks to a CSV and clean it from the memory.
Any suggestions will help.


texts:
Consider a vector $a \in R^n$.

I want to know how I can find analytically the solution of the following optimization problem: $x^* = argmin_{x \in R^n} f(x)$, where

$f(x) = ||x-a||_{2}^2 + \lambda ||x||_1$
$\lambda > 0$ and
$||.||_p$ is the p-norm in $R^n$.

Thanks in advance.


texts:
(First post here)
I am rather new to neural networks, having used Tensorflow for a couple months now, and am looking for some advice I have on an idea to improve the accuracy of my model. I am looking at sequences using 1d convolution layer with spatial dropout, and an LSTM to look for spatial relatedness of these features.
My idea was that since these sequences have different types of features with at different sequence lengths I might try sending my input sequence to multiple 1dconv layers with different window sizes and then merging them back together.

Would this be effective at all? Is there a way to some how link the layers so that the layers with the larger window would not learn patterns from layers of the smaller windows? Or should I just make the original convolution layer larger with more filters?

PS The my original idea was to pair each individual convolution with an LSTM on top of it, and merge these into another convolution with an LSTM on top of that, but it seems to be prohibitively complex on my computer.


texts: What to set in steps_per_epoch in Keras' fit_generator?

texts: I am replicating, in Keras, the work of a paper where I know the values of epoch and batch_size. Since the dataset is quite large, I am using fit_generator. I would like to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way?


texts: Which tool is good to collect tweets on 50 keywords over the last 5 years and then analyze them with the LDA algorithm or sentiment analysis?

texts:
I want to find tweets from the last 5 years to a topic. For this I decide for 50 Keywords (related to the main topic), where I want to find data on Twitter. I want to find out how the trend on the subject is, are there any opposing/negative opinions, are there recognizable patterns, ... (I am also unsure which possibilities I have to get something useful from the data)
I totally new to Data Science and never done it before.
I have found the following tool https://algorithmia.com/pricing and https://monkeylearn.com/ . Especially Algorithmia looks like it could help me. Are there other alternatives that could help me with this task? Since my programming skills are not very good and my data science skills are not existing, I thought it would be the easiest thing to buy a tool.
I am happy about any advice. :)


texts: 2 stage ensemble -- CV MSE valid in 1st stage but not in 2nd

texts:
I'm trying out a Kaggle competition, which puts me in the unusual position of being able to get feedback on my models' "true" performance (you can submit several predictions per day and they give you your score on a performance metric -- in this case MSE -- each time).

As such, I've been observing a situation I can't explain -- my cross-validation results for my first stage Random Forest model, constructed with sklearn in Python, are consistently accurate within 1 percentage point of the true MSE.

For example, my first  stage model will give a CV MSE of about .22 and the "ground truth" (actual) MSE will turn out to be about .23.

OTOH, my 2nd stage Random Forest, which is a different type of RF constructed in R, gives way "better" (smaller) training and cross validation MSE errors, like 0.05, but consistently turns out to have a much higher true MSE, like .23. In the 2nd stage Random Forest I have the predictions from the first stage and a few additional engineered features as regressors.

Is there a statistical explanation for this?

For example is it the result of some sort of recursion from stacking the models? If so, given that most winners of these Machine Learning contests are stacking many models, what's the appropriate way to do so that would avoid this statistical problem?


texts: How to form Hessian matrix in BFGS Quasi-Newton Method

texts: I came across this link. In BFGS Quasi-Newton Method, a Hessian matrix is used in weight updation. Is there any resource where I can find how this hessian matrix was obtained along with a clear description of the process, as to why Hessian matrix has been taken? I could not understand the wiki article.


texts: Deciding on the number of components in PCA

texts:
I have been running my model several times now. Each time i get different results based on what number i put in my PCA component number range (I used raw numbers in the code instead of the range function).

If i put the range from 1 to the max_number (e.g.100) of components i get certain accuracy, lets say 60%, and the component number chosen is 80. So 60% at 80 components.

Now if i repeat the run with a range from 1 to 79, i get accuracy 62%, with number of component chosen as 45

If i run the whole thing again, while choosing range from 1 to 100 separated by 10 (instead of 5, or 1), e.g. range(1, 100, 10), I get a different accuracy as well.

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

So what should i do?

Should i run the analysis with component range 1 to max separated by 1 (e.g. range (1,max), and then each time i get a chosen component number i should investigate the series below it?
Can someone help please?

Here is my code

# Search for the best combination of PCA truncation
# and class reg (LogReg).

from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(random_state=42, class_weight= 'balanced', max_iter=5000)
pipe_logreg = Pipeline(steps=[('pca', pca), ('logreg', logreg)])

# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_logreg = [{'pca__n_components': [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100]},
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
                    'logreg__penalty':['l2'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['newton-cg', 'lbfgs', 'sag'],
                    'logreg__multi_class': ['ovr', 'multinomial', 'auto']},
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
                    'logreg__penalty':['l1'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['liblinear', 'saga'],
                    'logreg__multi_class': ['ovr', 'auto'],
                    }]

clflogreg = GridSearchCV(pipe_logreg, param_grid =parameters_logreg, iid=False, cv=10,
                      return_train_score=False)
clflogreg.fit(X_balanced, y_balanced)

# Plot the PCA spectrum (logreg)
pca.fit(X_balanced)

fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')

ax0.axvline(clflogreg.best_estimator_.named_steps['pca'].n_components,
            linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))

# For each number of components, find the best classifier results
results_logreg = pd.DataFrame(clflogreg.cv_results_) #(Added _logreg to all variable def)
components_col_logreg = 'param_pca__n_components'
best_clfs_logreg = results_logreg.groupby(components_col_logreg).apply(
    lambda g: g.nlargest(1, 'mean_test_score'))

best_clfs_logreg.plot(x=components_col_logreg, y='mean_test_score', yerr='std_test_score',
               legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')

plt.tight_layout()
plt.show()


texts: Applying bayesian methods to a simple neural network

texts:
This is a really simple neural network with backprop. If one had to apply bayesian "inferences" to update the weights and biases, what would change in the code.

#Forward Propogation
hidden_layer_input1=np.dot(X,wh)
hidden_layer_input=hidden_layer_input1 + bh               # linear transformation
hiddenlayer_activations = sigmoid(hidden_layer_input)     # non-linear transformation
output_layer_input1=np.dot(hiddenlayer_activations,wout)
output_layer_input= output_layer_input1+ bout             # linear transformation
output = sigmoid(output_layer_input)

#Backpropagation
E = y-output
slope_output_layer = derivatives_sigmoid(output)
slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)
derivative_output = E * slope_output_layer
Error_at_hidden_layer = derivative_output.dot(wout.T)
derivative_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer
wout += hiddenlayer_activations.T.dot(d_output) *lr
bout += np.sum(derivative_output, axis=0,keepdims=True) *lr
wh += X.T.dot(d_hiddenlayer) *lr                          # update weight
bh += np.sum(d_hiddenlayer, axis=0,keepdims=True) *lr     # update bias

If my understanding is correct of using bayes method to derive the weights and biases, it would help in getting to those values faster, leaving behind unnecessary loops for weights which would not converge.

Apart from that, coding and observing it is the best bet to get a real perception.

Would like an understanding.


texts:
I have a corpus of data

Class   Tweet
-1  toxic phenol ingredient in vaccines  the detrimental effects of injected phenol  have not been fully evaluated
-1  doctors give flu shots to pregnant women despite evidence of harm to fetus
-1   hearus autism and the mmr vaccine  the most diabolical medical scandal of the century       cdcwhistleblower  vaxxed
-1  rt   how to naturally detox from mandatory vaccine injections     hearus  cdcwhistleblower
-1  rt   the removal of vaccine exemptions forces parents who know that vaccines injure and sicken their children into coerced ha

I'll be doing some supervised learning on the text.  I put the text through a pipeline as follows:

selector = SelectKBest(chi2, k = K)

clf = SVC(kernel = 'linear')

p = Pipeline([('vect', CountVectorizer()),
                    ('tfidf',TfidfTransformer() ),
                    ('feat',selector),
                    ('clf',clf)])

SelectKBest will return K of the best features.  I suppose then that those will be used to train the model.

Do I need to predict new tweets on the same features?  Or can I just leave the pipeline as is and use CV to find the best K?


texts: Plotting Gradient Descent in 3d - Contour Plots

texts:
I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost in 3d to visualize the level sets on the contour plots and the cereal bowl function.

House Dataset with 3 parameters (1's, bedrooms, Sq.ft) to predict prices having shape $(100000, 3)$ and $y(100000, )$. The goal is to look at the cereal bowl function in 3d and look at how the gradients are converging.

References: Gradient descent implementation in python - contour lines:

def compute_cost(X, y, theta):
    return np.sum(np.square(np.matmul(X, theta) - y)) / (2 * len(y))

def gradient_descent_multi(X, y, theta, alpha, iterations):
    theta = np.zeros(X.shape[1])
    m = len(X)
    j_history = np.zeros(iterations)
    theta_1_hist = []
    theta_2_hist = []
    for i in range(iterations):

        gradient = (1/m) * np.matmul(X.T, np.matmul(X, theta) - y)

        theta = theta - alpha * gradient

        j_history[i] = compute_cost(X,y,theta)
        theta_1_hist.append(theta[1])
        theta_2_hist.append(theta[2])

#         J_history.append(compute_cost(X,y,theta))
#         print(J_history)

#         grad_plot.append(theta)

    return theta ,j_history, theta_1_hist, theta_2_hist

theta = np.zeros(2)
alpha = 0.1
iterations = 100

#Computing the gradient descent
theta_result,J_history, theta_0, theta_1 = gradient_descent_multi(X,y,theta,alpha,iterations)

Theta 1:
[15.651431183495157,
 28.502297542920118,
 39.0665487784193,
 ...
 105.78644212297141,
 105.882701389551,
 105.97741737336399]
Theta 2:
[14.713094556818124,
 26.640668175454184,
 36.29642936488919,
 ....
 59.1710519900493,
 59.07633606136845]
Cost array:
array([185814.55027215, 149566.02825652, 120605.70700938,  97414.66187874,
            78807.39414333,  63853.50250138,  51819.24085843,  42123.5122655 ,
            34304.44290442,  27993.78459818,  22897.16477958,  18778.74417703,
           ....
             1257.38095357,   1257.13475353,   1256.89643143,   1256.66572779,
             1256.44239308,   1256.22618706,   1256.01687827,   1255.81424349,
             1255.61806734,   1255.42814185,   1255.24426618,   1255.06624625])


texts: Classification - get some label value to check how close to another class (Python)

texts:
I am doing text classification in python with 3 alghoritms: kNN, Naive Bayes and SVM. I have 3 classes - easy, medium and hard. The accuracy is quite fine. Is there a way to check for new text its exact value? After label encoding 0 is easy, 1 is medium and 2 is hard. So base on new text for example it is classified as medium but I want to know how close it was to easy/hard. Some of my code snippets:

Train_X, Test_X, Train_Y, Test_Y = model_selection.train_test_split(
    df['tokens'], df['class'], test_size=0.3, random_state=42)

Encoder = LabelEncoder()
Train_Y = Encoder.fit_transform(Train_Y)
Test_Y = Encoder.fit_transform(Test_Y)

Tfidf_vect = TfidfVectorizer(max_features=35)
Tfidf_vect.fit([' '.join(arr) for arr in df['tokens']])
Train_X_Tfidf = Tfidf_vect.transform([' '.join(arr) for arr in Train_X])
Test_X_Tfidf = Tfidf_vect.transform([' '.join(arr) for arr in Test_X])
Naive = naive_bayes.MultinomialNB()
Naive.fit(Train_X_Tfidf, Train_Y)
predictions_NB = Naive.predict(Test_X_Tfidf)
print(round(accuracy_score(predictions_NB, Test_Y)*100, 2))

Now when I use Naive.predict() I get 0, 1 or 2. Is there a way to get the EXACT value for example 0,5897237489 which is 1 but I see that it is closer to 0 than 2

If it is not possible with NB, I used SVM and kNN as well:

SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto')
SVM.fit(Train_X_Tfidf, Train_Y)
predictions_SVM = SVM.predict(Test_X_Tfidf)
print(round(accuracy_score(predictions_SVM, Test_Y)*100, 2))

KNN = KNeighborsClassifier(n_neighbors=3)
KNN.fit(Train_X_Tfidf, Train_Y)
predictions_KNN = KNN.predict(Test_X_Tfidf)
print(round(accuracy_score(predictions_KNN, Test_Y)*100, 2))


texts: The learning_rate in TensorFlow code is the sum learning rate of a batch or the learning rate of a data?

texts:
import tensorflow as tf
batch_size = 10
learning_rate = 0.001
tf.train.AdamOptimizer(learning_rate)

So the question is:
The learning_rate of a data is 0.0001 or 0.001 ?
The sum learning_rate of a batch is 0.01 or 0.001 ?


texts: I have a value associated with each US state (let's pretend it's the average temperature in January for each state). I want to display this data as a heat map of the United States. To be clear, it would be a map of the US with each state having a color from a color gradient that corresponds to a quantitative value. I don't know what this plot type is called, so I'm having trouble figuring out how to do this in MATLAB or another program, but does anyone know an easy way to do this? I have a vector of data (of length 50) and just need to make a US map plot.