texts: A/B Test: Determine if a Lift in Measured Sales is Significant

texts:
I am no stats expert so any assistance will be greatly appreciated.

I’m working on a post-campaign analysis to determine if any measured sales lift (sales lift per audience) is significant between test and control treatment groups.   From what I have been reading, I believe you could potentially use an Anova or Ancova to run a post-campaign analysis to determine if any measured sales lift is above or below a standard deviation or margin of error?

My question is, does this make sense?  I’m never run an ANOVA is SAS is there any sample code one can share?


texts: Changing Timestamp format for Date-Time in Excel/Pandas/Python?

texts:
I have a excel data with time stamp format like this "2019-06-10T14:05:00+05:30"
[YYYY-MM-DDTHH:MM:SS], I want it to be converted into "2019-06-10 14:05" [YYYY-MM-DD HH:MM],

Is it possible to convert into required format using Excel or Python Libraries?
or any other way?

Thank you!


texts: Binary classification of partially labeled data

texts:
One of my friends was asked this question in an interview. A clue/restriction is given: Do NOT use semi-supervised learning techniques.

Suppose you have a binary classification problem: There are ~10K labeled points and ~1M unlabeled points. What are the steps in solving this problem?


texts: SGD Classifier partial fit throws ValueError on second iteration

texts:
I am currently training a nlp model with labelled data to find area in an address. Using SGDClassifier's partial_fit function, I am trying to split training data into smaller batches due to insufficient memory and have encountered the mentioned error.
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import SGDClassifier
from sklearn_pandas import DataFrameMapper

# train.shape = (300000, 2)
# train.columns = (['address', 'area'])

# data preprocessing function
to_vect = Pipeline([('vect', CountVectorizer()),
                    ('tfidf', TfidfTransformer())])
full_mapper = DataFrameMapper([
            ('address', to_vect)])
full_pipe = Pipeline([('mapper', full_mapper),
                      ('clf',SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, random_state=42))])

# pipeline and fit
def partial_pipe_fit(pipeline_obj, df, class_labels):
    X = pipeline_obj.named_steps['mapper'].fit_transform(df)
    Y = df['area']
    pipeline_obj.named_steps['clf'].partial_fit(X,Y, classes = class_labels)

# batch size
n=int(len(train)/20)
class_labels = np.unique(train['area'])
for i in range(0, len(train),n):
    partial_pipe_fit(full_pipe, train[i:n+i], class_labels)
    print("{}%".format(n/len(train)))

It returns this error
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-c657333c8b0f> in <module>
     27 class_labels = np.unique(train['POI'])
     28 for i in range(0, len(train),n):
---> 29     partial_pipe_fit(full_pipe, train[i:n+i], class_labels)
     30     print("{}%".format(n/len(train)))

<ipython-input-15-c657333c8b0f> in partial_pipe_fit(pipeline_obj, df, class_labels)
     20     X = pipeline_obj.named_steps['mapper'].fit_transform(df)
     21     Y = df['POI']
---> 22     pipeline_obj.named_steps['clf'].partial_fit(X,Y, classes = class_labels)
     23
     24 # batch size

/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_stochastic_gradient.py in partial_fit(self, X, y, classes, sample_weight)
    696                                  learning_rate=self.learning_rate, max_iter=1,
    697                                  classes=classes, sample_weight=sample_weight,
--> 698                                  coef_init=None, intercept_init=None)
    699
    700     def fit(self, X, y, coef_init=None, intercept_init=None,

/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_stochastic_gradient.py in _partial_fit(self, X, y, alpha, C, loss, learning_rate, max_iter, classes, sample_weight, coef_init, intercept_init)
    511         elif n_features != self.coef_.shape[-1]:
    512             raise ValueError("Number of features %d does not match previous "
--> 513                              "data %d." % (n_features, self.coef_.shape[-1]))
    514
    515         self.loss_function_ = self._get_loss_function(loss)

ValueError: Number of features 446 does not match previous data 456.

The error occurs on the second iteration, I've followed the documentation where it mentions classes are to include all possible classes. How can I solve this issue?


texts: When is a Neural networked considered to be overfitted?

texts:
When is a neural network considers to be overfitted to a specific task.

I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting..

I use CV for training.
first i split the data into train/test => 90/10.
and within the training set ,i do another 90/10 split, which are those data I use for the training.

I then use the model for processing the complete dataset.

CV would state how much i am overfitting (I am getting minor error, deviating with 2%).. But since i am using most of the data for training? am I then creating an overfitted neural network for this dataset.

I am currently having 230043 training samples,  107145 total params, and validates on 25561 samples.  am I overfitting?


texts: Full Doc2Vec Implementation/Repdoduction in TensorFlow?

texts:
I'm looking to reproduce the doc2vec, i.e. paragraph vector approach by Le & Mikolov.
Is anyone aware of a full script using Tensorflow? In particular, I'm looking for a solution where the paragraph vectors of PV-DM and PV-DBOW are concatenated.
So far, I found:

GitHub #1: Only PD-DM, also I'm not sure how to run this script?

GitHub #2: Unclear what it is (PV-DM?)

GitHub #3: Looks like PV-DM?

There seems to be no solution that relates to the original IMDB data, and covers both PV-DM and PV-DBOW. I am aware of a few gensim scripts, but would like to know whether there's a TF solution out there.


texts: Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

texts: I understand the concept of backpropagation in standard neural networks and  backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of the maths behind it all, but what I don’t understand is why this affects the earlier timestep in particular? Since, the parameters (weights) in the RNN are all shared between timesteps, why is it that the earlier timestep is more affected? Wouldn’t they all be affected since they all share the same badly optimized weight which is never updated due to the many small terms in the multiplicative product which produces $\frac{\partial E}{\partial w} $? I feel like I'm totally misunderstanding something here. Many thanks


texts: Language translation with convolutional neural network

texts:
Many examples of language translation neural networks:

  "the cat sat on the mat" -> [model] -> "le chat etait assis sur le tapis"

use RNN, and in particular LSTM. See for example Sentences language translation with neural network, with a simple layer structure (if possible sequential) and A ten-minute introduction to sequence-to-sequence learning in Keras
.

Are there (not too complicated) attempts of doing language translation with CNN only and no RNN/LSTM?

Would you have an example in Keras?


texts: One-class classifier for time series data classification

texts:
My problem is different from the common time series data problem. What I need to do is check if future time series data is in accord with previous time series data I already consider to be correct.

In short, I need a one-class classifier applied to time series data, which have variable length (going from 110 to 125 points).

Does anyone know any lib that might help me with this task? I usually use Python+scikit-learn, but I'm open to use any other configuration.


texts: Are there free cloud services to train machine learning models?

texts:
I want to train a deep model with a large amount of training data, but my desktop does not have that power to train such a deep model with these abundant data.

I'd like to know whether there are any free cloud services that can be used for training machine learning and deep learning models?

I also would like to know if there is a cloud service, where I would be able to track the training results, and the training would continue even if I am not connected to the cloud.


texts: is it a good idea to take the derivative or integral of some features and add them as new features in machine learning?

texts:
I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column and the dataset is a time series dataset so they are monotonic. would it make sense to calculate the derivative or integral and add it as new feature ?

as an example let's say I have speed and acceleration as features, would it make sense to add the jerk (which is the derivative of the acceleration) and the snap(which is the derivative of the jerk) as new features ? also maybe the integral of the speed which would give the displacement I think?

the goal is let's say 2 features are not enough and we want to produce more features, is it wise to add the derivative or integral as a new feature? or is it a bad idea?

I also want to know whether the correlation between the derivative and integral according to timestamp and the feature that I derivated from would be high if I do this and is it bad or good if I make new features that correlate with others in my dataset


texts: A Proper Outlier Detection For the Attached Figure


texts: Plotting three lines on the same plot (with four hr frequency)

texts:
I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used:

 data = pd.read_csv('p1.csv')
 df1 = pd.DataFrame(df1, columns=['date', 'time', 'x1', 'x2','x3'])
 data['date_time'] = pd.to_datetime(data['date'] + ' ' + data['time'])
data['date_time'] = pd.to_datetime(data['date_time'], format='%m/%d/%Y %H:%M:%S')
data.set_index('date_time',inplace = True)
plt.plot(data.index, data.x2)

plot graph :

Can anyone suggest me a solution for this?

csv file after reading it:

i want to plot like this but withdate


texts: 4 Class Classification - Machine Learning Model

texts:
I have a data set which contains nearly 150 features and 60k data. And my target feature is continuous variable represents hours. I divided this period into 4 categories of user engagement (4 ranges of hours). Implement GA with SVM, GA with logistic regression, Random forest, GA with KNN with suitable normalization of data wherever required. Used GA for best feature subset selection.

All the algorithms gave similar results of around 46% accuracy ( for nearly balanced test set).

Note: Training is also done on a balanced data set. I am wondering where am I gone wrong?

I believe I went wrong somewhere in input to target mapping. Could anyone please confirm that categorization of the continuous target variable (hrs) into 4 sets are reasonable?


texts: What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the document.


texts: Multi-label classification with missing labels

texts:
I have a neural network that generates a vector that represents the class probabilities. Since it is a multilabel classification problem, I'm supposed to train the network using sigmoid + binary cross-entropy to calculate class probabilities independently but I only have access to one correct class i.e. my target vector is a one-hot vector where the vector values has one for the correct class and zero everywhere else.

Currently, I'm training the network with softmax + cross-entropy but I was wondering if there is another way to train.


texts:
I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying different models.

After working with some results and reading more and more about LSTM, it seems like this approach is far superior to anything else I've tried (across multiple datasets). I keep thinking to myself, "why/when would you not use LSTM?". The use of the additional gates, inherent to LSTM, makes perfect sense to me after having some models that suffer from vanishing gradients.

So what's the catch with LSTM? Where do they not do so well? I know there is no such thing as a "one size fits all" algorithm, so there must be a downside to LSTM.


texts:
If I have some data regarding the occurence of an event on a certain date and some other variables regarding it (think fe.: I have data on which dates it rained, and some addtitional data like temperature, atmospheric pressure etc.), which is the most appropriate model for predicting on which day the event is going to happen again? Or, to be more precise, I'd like to predict the frequency of said event, to know in how many days it's going to occur again. I mostly use Python with the numpy, sklearn libraries, and I'm interested which of its models fits my use case best.

Thank you!


texts:
I have been working on a project and we were trying to convert a PSD file to a HTML for web applications as well as a Layout XML for android. We worked our way to generate basic skeletal html/xml but hit a wall for complex scenarios such as identifying separate divs and components.

Our initial approach was to standardize the PSD and get metadata about each component from PSD but due to it's limitations we could only add metadata as a prefix to the name of each layer or component for example : If PSD has a component which is suppossed to be button we would name it btn_submit and convert this component into button in HTML/XML based on btn suffix. But this was not scalable and seems more like Expert Systems. So our team decided to move to Machine Learning.

We are considering supervised learning as our first approach so I would like to know which algorithms can guide me to achieve this and is it better to use Expert System instead of Machine Learning in this case?


texts:
I have soccer data with a time series index. 30 seconds interval. So, 194 rows for 90+ minutes per game. I have 1500 games.
The dataframe has the following information.

Home/Away:

•   Goal Total.

•   Shots Total.

•   Dangerous Attack Total and.

•   XG(Expected goal per game) given at the start of the match which needs to be adjusted per section.

I am required to build a time series model to predict time t+1 (say Goal score).

Given that for each variable the graph shows the highest spike in the later stage of the time, Is there any machine learning algorithms that can learn with such a small dataset?

Sample Data

g = pd.DataFrame({'TimeSlot':[0, 30000, 60000, 90000, 120000, 150000, 180000, 210000, 240000, 270000, 300000],
    'xG_A':[1.5]*11, 'xG_B':[1.2]*11, 'A_DAT':[0,1,2,3,4,4,5,5,6,6,7],
    'B_DAT':[0,0,1,1,2,2,2,3,3,3,4],
    'A_ST':[0,0,1,1,2,2,2,3,3,3,3], 'B_ST':[0,0,0,1,1,1,1,2,2,2,2],
    'A_GT':[0,0,0,0,0,0,1,1,1,1,2], 'B_GT':[0,0,0,1,1,1,1,1,1,1,1]})
g.set_index('TimeSlot', inplace=True)

The assumption is only one Goal/Shot can be scored/taken per Timeslot. So the same data per period will look like...

f= pd.DataFrame({'TimeSlot':[0, 30000, 60000, 90000, 120000, 150000, 180000, 210000, 240000, 270000, 300000],
    'xG_A':[1.5]*11, 'xG_B':[1.2]*11, 'A_DATP':[0,1,1,1,1,0,1,0,1,0,1],
    'B_DAP':[0,0,1,0,1,0,0,1,0,0,1],
    'A_STP':[0,0,1,0,1,0,0,1,0,0,0], 'B_STP':[0,0,0,1,0,0,0,1,0,0,0],
    'A_GTP':[0,0,0,0,0,0,1,0,0,0,1], 'B_GTP':[0,0,0,1,0,0,0,0,0,0,0]})
f.set_index('TimeSlot', inplace=True)

***ARIMA(2,1,0) with train, test = X[0:size], X[size:len(X)]
gives MSE:0.323 for g and MSE: 0.291 for f.***

Questions

Which dataframe should I use?

I would really appreciate if someone starts off with a few python codes...


texts: Q values loss per episode and mean absolute error

texts:
I am new to deep reinforcement learning! I am following this code for my adaptation problem (doing actions)
https://github.com/jaromiru/AI-blog/blob/master/CartPole-DQN.py
I am wondering how I can evaluate the training, I already got the average rewards, but how can I get the average Q values, loss per episode, and average absolute error. to evaluate my agent please!
I will be grateful if you can help me!


texts: What should be the value of non-rated field when finding cosine similarity

texts:
I am working on a very basic book recommender system. I want to know what to do with the fields which aren't rated by the user when finding cosine similarity, should we ignore them and calculate only with the rated fields or should we mark them 0.

The book I am following says to exclude the fields since it will give wrong interpretation in case of Euclidean and Pearson Correlation but in case of Cosine Similarity, it makes all the non-rated fields to 0.

Can someone explain why is it needed to make the non-rated fields to 0 only for Cosine and not others or is there a different way to do it.
(I know how making fields 0 in Euclidean and Pearson affect the output but not sure of cosine)


texts: How to choose between discounted reward and average reward?

texts:
How to select between average reward and discounted reward?

And when average reward is more effective in comparison with discounter reward and when vice versa is correct?
Is is possible to use both of them  in a problem? Because as I understand the RL reward  is based on average reward or discounted future reward, but I think this paper use the discounted and average together. Is it correct: we use discounted future reward in order to training and average reward in test and evaluation? What is wrong in my understanding?

In this picture, figure 2 of the paper "Playing Atari with Deep Reinforcement Learning":

The authors report the "average reward". However, in the same paper, the authors also mention "discounted reward". So, I'm confused. What is the difference between discounted reward and average reward?


texts: Multiple target prediction for a Regression Problem

texts:
I am doing my master's thesis on Indoor positioning system using BLE rssi. I have prepared a dataset with 4 features (rssi values from 4 ble rssi scanners). Target columns are x-coordinate and y-coordinate.
I want to predict both x and y coordinate at the same time. I can't find a way to do that. Crrently what I am doing is making 1 model for predicting x-coordinate and doing same for y-coordinate. The problem here is that model is learning y-coordinate better with MSE close to 1.5 while in learning x-coordinate the MSE is 2.5.
The regressors that i am using are: Linear Regression, kNN regressor, SVM regressor and random forest regression.
How can I have 1 model for both x and y and then single MSE using both x and y?


texts: Determine statistically whether new product cannibalise old product by using data

texts:
Assume that $A$ is a cab company which offers online cab booking through their standard account.

Recently, the company launched a pre-paid premium account with features such as discounted rides,special offers, guaranteed pick up and low waiting time.

A survey is conducted to public and it shows that the response to the new premium account is encouraging.

  Question: Does the encouraging response stem from new customers or existing customers?

We are given a data consisting of time (booking is placed), pick-up and drop-off location, journey fees, ride type (premium or standard), customer demographic and pricing information.

I am totally lost here. What can we do to answer the question above? Any hint is appreciated.

If I were asked to determine whether customers have any fixed budget to transport, how can I solve it using data above?

My plan is to split the spending into monthly sales, then calculate their means. Let's say we have $12$ sample means and sample variances.
Since the sample variances might not be equal, I employ Welch ANOVA test.
Is this sufficient?


texts: keras UserWarning: The input 1303 could not be retrieved. It could be because a worker has died

texts:
I am trying to train an autoencoder CNN on Google Colab using keras. I have mounted my Google Drive which contains all training data. The training uses six workers and the data is loaded by the following custom generator class:

class ImageDataGenerator(keras.utils.Sequence):

    def __init__(self, filepaths, batch_size, shape, shuffle=True):
        self.shape = shape
        self.batch_size = batch_size
        self.filepaths = filepaths
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        return int(np.ceil(len(self.filepaths) / self.batch_size))

    def __getitem__(self, index):
        from_index = index * self.batch_size
        to_index = min(len(self.filepaths), (index + 1) * self.batch_size)

        filepaths_temp = self.filepaths[from_index : to_index]

        images = load_images(filepaths_temp) / 255.0

        return images, images

    def on_epoch_end(self):
        if self.shuffle:
            np.random.shuffle(self.filepaths)

My problem: Sometimes when I start training I get this:

Epoch 1/100
/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 1303 could not be retrieved. It could be because a worker has died.
  UserWarning)
1599/1599 [==============================] - 2461s 2s/step - loss: 0.0239 - val_loss: 0.0251

Epoch 00001: val_loss improved from inf to 0.02512, saving model to /content/drive/...
Epoch 2/100

Apart from the warning epoch 1 seems to finish successfully. But after the last line is printed, nothing happens anymore. Epoch 2 never starts.

I suspect the mounted Google Drive filesystem is the problem, because I had problems with Google Drive timeouts in the past. My Question is: Is there something I did wrong? Did one of the six workers die? Is there something I can do to restore the workers after the first epoch? After all, it seemed to work fine for the first epoch.


texts: Approach to creating a user profile in music web application

texts:
I am working on a use case, and I'm unsure of the best way to proceed: in order to analyze the behavior of users of a web-based music application, we retain all songs each has played since 2009. We store this information in flat files, each containing the songs played in a day. Each file contains 50M of lines, and we have  19M users. Our entire song catalogue consists of 35M tracks.

The format of these files is as follows:

id-user | country | id-artist | id-track

Question: I would like to represent each user by the songs he or she has played; this profile would be used by the production site. Does anyone have a suggestion for what would be the best way to process the whole chain?


texts: Predict set elements based on other elements

texts:
Sorry if this has been answered before but could someone help me with solving the following problem:

Each symbol in a dataset has a set of labels. Given a set of labels how can we predict more labels for that set?

Or to attempt a more formal wording: let a set of sets $S = \{ s_1, s_2, s_3,  ... \} $ where $ s_i \subseteq L $ and $ L = \{ l_1, l_2, l_3,...,l_m\} $ where $L$ is the finite set of labels. Given a set of labels $Y = \{ y_1 ,..., y_n \} $ where $Y \subseteq X $ what is the probability $ P(r = l_i) \forall l_i \in L$ so that $Y \cup \{r\} \in S $.


texts: How can causal inference be used with machine learning?

texts:
I am wondering how Causal inference is being used with machine learning and specially where in the data science project ? I have been searching for an answer and I've came to the conclusion that Causal inference can be used after the modeling phase in order to confirm some correlations between variables and the target/outcome.

For example if the model has a good accuracy and gives you a high correlation/association between input A and the target B you may want to perform a causal inference to validate that A has an effect on B.

I would like to know if my understanding is correct and also know if there is other application of Causal inference with Machine learning.


texts:
I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another csv.
In Jupyter Notebook
Thank You!


texts: Why does the parent connection matter during a NEAT crossover when both connections are identical?

texts:
I've been reading through NEAT (NeuroEvolution of Augmenting Topologies)'s whitepaper, and one thing is particularly confusing:

Innovation numbers are supposedly used to track the origin of a mutation, so that crossing over is easily done. From what I understand, two identical connections should always have the same innovation number, because their numbers are indexed. However, according to this illustration:

when the two genomes being crossed over share connections with the same innovation number, the resulting innovation is taken from the more fit parent (or random if they are equal fitness).

If the two connections are identical (they have the same innovation number) then why does it matter which parent it comes from, since the connection is the same either way? Or is there something about them that is not the same and I am understanding this wrong?


texts:
I have a classification task for people with 3 categories. I want to apply machine learning for that. I have 10 sources of data, which have the same fields (say 4: age, job title, a number of organizations, a number of followers). Data is incomplete, some fields can be missing in some profiles. The training set is limited (say, 300 examples).

I have two strategies for feature engineering, and I don't know which one to use.

Expand features: take 40 features (Profile 1 age, Profile 1 job title, ..., Profile 10 age, Profile 10 job title).
Compact features: take 4 features, and apply some heuristics to merge the values from different profiles. Say, take age and job title which occur most frequently, take a maximum number of organizations, take a sum of numbers of followers.

What strategy is generally used to give best results and why?


texts: Choosing a learning rate interval for linear separation with a perceptron

texts:
I'm having an issue with an exercise regarding the use of a perceptron for the linear separation of two sets M0 and M1. The question at hand is, how to go about finding an interval for a reliable learning rate of the perceptron.
The two sets are:

Or as a graph:

As can be seen from the graph, the x distance between the two sets is 1.
How would you go about finding an interval for a reliable learning rate? I'm guessing the lower limit for the learning rate would be close to zero but i'm having issues defining the maximum reliable learning rate without creating a model first and trying out different learning rates.
Thank you in advance.


texts: Product classification on multiple levels

texts:
I have to assign UNSPSC for each of my products. For the uninitiated, this means I have product description for each of my product, and I have to assign each an 8 digit number, where level 1 determines first 2 digits, level 2 determines the next 2 and so on...
I am able to get predict the level 1 number that is of the form XX000000 for each product in my test data by learning from the training data using the random forest classifier. My question is- to get to next levels, eg level 2, how do I ensure that my program not just learns from the product description but also level 1? How do I give it multiple columns as input?


texts: Help me understand how word-as-vector representations are constructed

texts:
Let's suppose I have a big list of words. I want to turn this list into a vector space of dimension $N$ such that each word is a vector in this vector space. But I have no idea how to go about with that. Some questions:

Is the list enough? For each element of the list, do I need $x$ example sentences also?
How does the computer deduce the dimensions of the vector space from the list/corpus?
Is there a way to figure out whether the dimensions of the vector space correspond to something in English?


texts: Can I create pretrain model with tensorflow?

texts: I takes a long time for train neural network model. It have to train every time when I run code. If I get high accuracy from training , Can I use same training model with another code without new training anymore ? How to do?


texts: How to do encode this target vector containing strings

texts:
Consider a target vector like ["x", "x", "y", "z", "z", "z", "x", "y", "y", ...] containing the names ("x", "y", "z", ...) of some writers. For examples, assume that we have some quotes and we want to predict who has quoted that.
I want to encode these target names to numerical values. If I use frequency encoding, then, if two writers appear the same number of times in the list, they will get the same value. This is not what I want because in this case, a machine learning algorithm considers those two different persons (labels) as one person (label). If I use ordering then I will have a huge difference in the labels. Because I actually have about 80,000 different authors. "Get dummies" also doesn't work because generated data frame doesn't fit into memory. So what is the best way to encode this target column in a data frame?
My second question is, do you use multi-class classification for this task? As I said there are 80,000 different writers so there will be 80,000 labels and each label appears, in average, about 50 times in the list. Some labels appear only one time and some about 400 times!!.
In general, what supervised learning algorithm will you use for a task like this?


texts: Match a two items from two different receipts

texts:
I have two different invoices or receipts. One is a Purchase order one is something like a receipt(acknowledgement).

Suppose I have ordered(PO) Wine:

White Wine
Red Wine
Rose Wine

And I receive the acknowledgement as:

Wine Red Jacobs Creek
White Wine
Winter's Hill Estate Dry Rose

I want to match the strings (items) in the Purchase Order and the Invoice.

Can you suggest me ways to do it.

I have tried vectorization using Count Vectorization Alg
Then have used distance measures to calculate similarity using:
'dice', 'rogerstanimoto', 'yule', 'hamming', 'jaccard', 'braycurtis', 'canberra', 'cityblock', 'correlation', 'cosine', 'euclidean', and 'minkowski'

The problem is the position of Words.

Red Wine is will not be similar to Wine Red. But that should not be the case.

I have tried Word2Vec Algorithm too but as this is not language technically just Nouns. It did not work.


texts: Could you please tell me why do we use a learning rate to move into the direction of the derivative to find the minimum? Why is it not good if you simply count it where is it 0?


texts: In classification task, is it possible that a truly classified data has a higher loss compared to a miscallisfied data?

texts:
Given a classifier using softmax, is it possible that say, for data point a which our model has correctly classified, has higher loss compared to data point b which is misclassified by our model?
If yes, please bring an example. If not, please prove.
================================================================
Update: I forgot to mention consider cross-entropy loss.


texts: How are the findings learnt from the data set are generalized compared to Statistics?

texts:
I am new to data science/ machine learning world. I know that in Statistics we assume that a certain event/ process has some particular distribution and the samples of that random process are part of some sampling distribution. The findings from the data could then be generalized by using confidence intervals and significance levels.

How do we generalize our findings once we "learn" the patterns in the data set? What is the alternative to confidence levels here?


texts:
How do I remove outliers from my data? Should I use RobustScaler? I am aware I can use DecisionTree but I want to use XGBoost...
Please can you help me, This is a bit urgent, I am not sure how to do it, I have researched and seen previous question but it doesnt work well and was not helpful.
Thank you

Cheers


texts: What are the possible values of a filter in a CNN?

texts:
I am a trying to write a CNN from scratch in python but I am bit new to CNNs specifically the convolution layers as I am comfortable with the dense layers. I was reading Do filters have different weights for each input channel but I didn't completely understand and had a few questions. I wanted to confirm that if the input layer had 3 channels then for there to be 4 output channels you would need a total of 12 filters.

Also, can the weights(actually I am not completely sure that I know what the weights are, they are the values of the kernels right?) of a filter be a floating number as  I commonly see them being described as -1, 0 ,1.

And finally, are the weights of the kernels. initialised once or rather are they changed during back propagation, as I used to have the notion that only the bias and dense layer would be changed.

Thank you so much for you time, any help is greatly appreciated.

EDIT: I have 1 more question, but I didn't feel it appropriate to make a whole new question. Basically, if I have a filter that looks like [0,1]. The stride is 2. Now if it has to run over [0,1,2,3,4]. First it will match [0,1] with [0,1] then with [2,3]. But won't it stop at [2,3] and never look at 4. How do CNNs solve this problem. Do they add an extra run of [0,1] and [3,4] but this would count 3 two times. Again thanks for the help.


texts: Improving accuracy on highly imbalanced dataset

texts:
I need some suggestions to improve my model accuracy.

The training data shape is : (166573, 14)

It has all int and float columns. I have dropped claims_daysaway column as most of values are NaN and replaced Nan value with mean for tier column.

X_train = train.drop(['outcome','testindex','claims_daysaway'], axis=1)
y_train = train['outcome']

As the values were on different scale, I have used StandScaler() to standardize values.

This dataset is highly imbalanced.

  train['outcome'].value_counts()

0    159730
1      6843

I tried SMOTE for oversampling.

from imblearn.over_sampling import SMOTE
smt = SMOTE()
X_train, y_train = smt.fit_sample(X_train, y_train)
pd.value_counts(pd.Series(y_train))

1    159730
0    159730

Lastly, I fit model using XGBClassifier but when tried this model on testdata and submitted it, it gives only 60% roc_auc_score.

Please suggest how to handle imbalanced dataset better.


texts:
I have a 3D tensor on which I apply 2D convolutions. Sometimes, this 3D is padded both in width and height to have a fixed size. How could I apply masking (like with RNNs) so that the gradients associated with the weights of these padded cells don't get updates while backpropagating the error?

Thanks in advance.


texts: Optimizing parameters for a closed (black-box) system

texts:
I am working on a problem that involves finding optimal parameter values for a black-box system. This system consists of 6 inputs and produces a single output. I can compare the returned values to observed values to determine whether the system is well calibrated based on the parameter values I specified.

The main problem that I am having is that I do not know how these parameters are used in the black-box system to produce the output. The parameters may be correlated so changing the value of one parameter may affect the way the others behave in the system.

Since I do not know the function I am not sure how to optimize this problem efficiently.

Question: what are some methods for optimization when the function is unknown?

Edits:

The variable types are each a vector of real numbers but we can make some assumptions if it is is helpful.

The cost to run the black-box system is only time. Let's say that it takes 10 seconds to run the system but I happen to have five black boxes -- so if I can run my algorithm in parallel then I can cut down the run time. If the algorithm runs sequentially then I don't gain anything by having the extra boxes (expect perhaps choosing different starting positions).

For this problem I am interested in learning about how to solve this type of problem -- not trying to brute force my way to a solution. That is to say I am less interested in obtaining an answer and more finding out how I would obtain the the answer. I think it is easy to say that given sufficient resources (time, processing power) that simply generating many many random numbers and choosing the combination that gives the best result is the simplest (and probably worst) approach.


texts: How one can match similar sounding features of two items

texts:
I am trying to build simple app which will compare two products from provideid URLs based on their features. We will be showing the products in tabular form with columns being each product and rows being features. The problem here is that we want to accept URLs from different shopping websites. We are able to fetch features from given URLs and get them as key - value dictionary, with key being feature name ('RAM Size' for laptop) and value be feature value ('8GB'). Difficulty is how can we match different features names from different shopping sites with same meaning? Like how we can map 'RAM Size' from one shopping site to 'Memory size' from other shopping site? We want to build app for several categories which have varying features like a pressure cooker might have volumne specified in litres. How can we generalise such matching of features for different kinds categories?
Do we have to go through the route of natural language processing? Or we will be better off doing manual mapping, even though a bit tedious? Is there any NLP library which can ease NLP based implementation?


texts: How to predict class label from class probability given by predict_generator for testdata?

texts:
While using Keras' flow_from_directory method to train my model on a multi-class image classification problem, the predict_generator function gives the class probabilities.

So, my query is how to get the corresponding class-labels for those class probabilities?


texts:
In the SMOTE paper here, the authors present the logic for creating synthetic examples when some of the features are nominal and some are continuous (section 6.1, SMOTE-NC).

This example is provided:

  $F_1$ = 1 2 3 A B C [Let this be the sample for which we are
  computing nearest neighbors]    $F_2$ = 4 6 5 A D E    $F_3$ = 3 5 6 A
  B K    So, Euclidean Distance between $F_2$ and $F_1$ would be:

  $Eucl$ = $\sqrt{(4-1)^2 + (6-2)^2 + (5-3)^2 + Med^2 + Med^2}$

  Med is the median of the standard deviations of continuous features
  of  the minority class.    The median term is included twice for
  feature numbers $5: B→D$ and $6:  C→E$, which differ for the two
  feature vectors: $F_1$ and $F_2$.

The paper lacks explanation about why the nominal features should be affected by the continuous ones.

Can anyone provide such explanation? Did I miss it in the paper?


texts: Why is the code provided in this book mostly commented out?

texts:
I am new to text mining and have been playing around with the code provided in the book Applied Text Analysis With Python. I came to a problem with this specific part:

https://github.com/foxbook/atap/blob/master/snippets/ch08/oz.py

In this python script, most of the code is commented out. Is there a specific reason why someone would do that? What's the correct way to uncomment it?

I am using jupyter notebook, I pasted the code in a cell and deleted one by one each '#', but I think I messed up the indentation.


texts: Do you need to perform variables reduction for tree-based models?

texts:
I know for methods and linear regression, GLM, Logistic regression, we typically run through a lot of variable reduction methods, i.e, forward/backward/stepwise, univariate analysis; variable importance through RF, GBM.

I know include too many features in linear model lead to overfitting, but I don't know if this is true for tree-based models.

I noticed this while I was building a GLM for a company project and through it might be a good idea to build a GBM and compare both.

Does tree-based models like RF and GMB, Catboost require you to drop variables in order to achieve a good model? - Re-build model many times eventually with a small model with fewer variables.


texts:
For https://www.intel.ai/winograd-2/ , why use stride = 2 ?

Why need to transform input image pixels ?

Why this C++ implementation of winograd convolution does not require any input tensors transformation ?


texts: Train and Test Data Split for a Tweet Classification Task

texts:
I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a combined disaster dataset. However, I want to apply it to individual disasters separately (e.g, train on multiple disasters, apply to earthquake; train on multiple disasters, apply to flood). I have the combined dataset in one pandas DataFrame and individual disasters in separate DataFrames.
In this case, how should I split my train and test datasets? At the moment, I split my combined dataset and apply ML algorithms.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'], all_disaster['target'], test_size=0.33, random_state=42)

But, I feel that is not the proper way, as I want to apply it into individual disasters.


texts: How to join 2 neural networks in tensorflow?

texts:
Suppose that by using Tensorflow, I'm able to construct to 2 functions:

f(x ,w1)=y (The first model- a FFNN)

g(x, w2)=t  (The second model - an RNN)

where: x, y, w1 are respectively the input, output, and weights of the first model; the same goes for x, t, w2

My goal is to construct a hybrid model whose weights only depend on w1 as follows:

g(x, f(x,w1))=t

To achieve that, I think the solution is to make the computational graph consider the weights w2 as a function of x and w1.

This is where I'm stuck. How can I solve it?

P/s : I do agree that reimplementing RNNcell is also a solution, but I'm looking for a more witty one. Because the reimplementation demands more lines of codes and we know the motto 'the fewer code, the fewer bugs'. Furthermore, that solution makes it more complicated to use many useful methods like Dropout, Bidirectional


texts: Implementation of an Adaptive Normalization method

texts:
Referencing: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.718.9985&rep=rep1&type=pdf
I'm trying to wrap my head around the method described in this paper
I currently have 2 main questions:

In the equation for R[i] on page 4, does the top row become S[0] when i = 1? If so, what is S[0]? And if not, how should it be calculated then?

Table IV on page 5, could someone explain how the table is derived? As well as why the paper goes from a single sequence in Table III to multiple sequences in Table IV?


texts: How to use a predfined separate prune set in Weka (alternatives in R or Python also acceptable)

texts:
I recently ran into a problem that seems to have considerate local effects, in space and time in fact: Two objects that are close are likely yo be in a similar state (if they are in the same year). I'm interested in modelling the behavior over time, but many classifiers find rules that are based on locality.

Now I have taken some action to prevent that of course, such as downsampling the set to achieve maximum variance over x and y and testing on a set that has been selected on a year that is not in the train set. I do however suspect that some techniques (tree learners) could benefit from  a prune set that has been separated on the same grounds as the test set.

Now most algorithms separate prune sets on initialisation themselves, but I want to experiment with a user defined set. In the current setup, rules that have been generated on local effects, would contribute less to performance in a test run, but the prune routine should be less equipped to predict that behavior.

Does anybody have experience with this? Do you know how to practically do that? Or do you know of a good theoretical resource on the subject?


texts: Sampling labeled data for anomaly detection

texts:
I'm currently working on a project that requires the use of unsupervised anomaly detection, but I'm unable to find a relevant data set, so I'm considering the following option:

Assuming I have a data set $X$ of $m$ examples labeled using $K$ classes. Let $X(k)$ be the subset of $X$ where all examples are labeled as $k$, and $k_{max}$ be the larget class.
Can I use $X(k_{max})$ as a training set for an anomaly detector, whose task is to flag elements who weren't labeled as $k_{max}$, as an anomaly?
Using $p << [m - size(X(k))]$ of the remaining examples in $X$ for cv and test sets as the anomalous examples.


texts: Data wrangling for a big set of docx files advice!

texts:
I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (for me anyway) data science project, so I'm really in need of some wisdom on the best way to approach it.

Essentially I have a set (200+) of docx files that are semi-structured. By semi-structured I mean the information I want is organized into tables (it's a form, with different tables which contain different info to fill out), but unfortunately these tables are not consistently formatted. Sometimes after people enter data into them they accidentally hit backspace to stick the tables together. Or sometimes they accidentally break the tables apart, for example.

My first attempt used python-docx to extract the data using document.tables[0] etc. I could then pull this into a big python dictionary for each document. It was quite neat but hit a snag - the table formatting problem above.

I then used python-docx again and tried to use the headings of each table as a marker (picked them out using regex) for when a sub data set should begin or end, and iterated across all text in the document. This sort of works, and is more flexible, but picks up a lot of text from outside of the table which makes it difficult to manage and clean.

Anyway, I'm interested in how an experienced data scientist would approach the problem.

The end goal is to extract the data from one of these documents into an SQL database.

If you're interested in the problem, let me know and I can send you the template documents I'm working with and some samples. If it's helpful, I can also post the code I've written so far (haven't done so because it's long).


texts: High resolution satellite data for climate analysis

texts: i'm into a smart agriculture project. I need to get past info about climate in a specific region. Could you recommend satellite climate data with highest resolution?


texts: python - Will this data mining approach work? Is it a good idea?

texts:
I need to extract fields like the document number, date, and invoice amount from a bunch of .csv files, which I believe are referred to as "unstructured text." I have some labeled input files and will use the NLTK and Python to design a data extraction algorithm.

For the first round of classification, I plan to use tf-idf weighting with a classifier to identify the document type - there are multiple files that use the same format.

At this point, I need I way to extract the field from the document, given that it is X type of document. I thought about using features like the "most common numbers" or "largest number with a comma" to find the invoice amount, for example, but since the invoice amount can any numerical value I believe the sample size would be smaller than the number of possible features? (I have no training here, bear with me.)

Is there a better way to do the second part? I think the first part should be okay, but I'm not sure that second part will work or if I even really understand the problem. How is my approach in general? I'm new to this kind of thing and this was the best I could come up with.


texts: How to set weights in multi-class classification in xgboost for imbalanced data?

texts:
From this post, I know you can set scale_pos_weight for an imbalanced dataset.  However, for the multi-classification problem in the imbalanced dataset, I don't quite understand how to set the weight parameter in dmatrix.

How can I use XGBoost for an imbalanced dataset in a multi-classification problem?


texts:
Say I am training neural networks using a train set and set aside a validation set V. I obtain models h's after each epoch along with the validation losses(0-1 loss) $\hat{L}(h_1,V)$, $\hat{L}(h_2,V)$ ... if I use the early stopping rule suggested here(top answer).

Is the resulting $\hat{L}(h_*,V)$ an unbiased estimate of the true loss?
How can I bound the true loss using $\hat{L}(h_*,V)$ ?

I'm guessing no for the first one since the stopping rule depends on the partition of my data set.

Afaik the bounds that can be applied depends on the size of my hypothesis set and I'm not entirely sure if it's finite, countable or uncountable in this case.


texts: Features standardization - Multilayer perceptron

texts:
I have serious doubts concerning the features standardization done before the learning process of a multilayer perceptron.

I'm using python-3 and the scikit-learn package for the learning process and for the features normalization.

As suggested from the scikit-learn wiki (Tips on pratical use), I'm doing a features standardization with the Preprocessing module, which means that all my features will appear as standard normal specifications.

The problem is that in the showed example:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
# Don't cheat - fit only on training data
scaler.fit(X_train)
X_train = scaler.transform(X_train)
# apply same transformation to test data
X_test = scaler.transform(X_test)

The standardization is done first on the training set, and then, after that, it's applied on the test set.

Which means, that the distributions parameters (mean, st.deviation) will be estimated only from the training set.

Now, because I need to save my multiple datasets in a database, on which I will perform the hold-out sampling for the training-test folds, I have done a preprocessing on all my instances at once, because I don't want to do that at run time execution.

It's clear that in this way the distributions parameters will change and the results will be different.

Now, Someone knows the differences between the two approaches? Are there issues doing the preprocessing at once on all the dataset?


texts:
There is something wrong with the following explanation regarding this figure:

The data densities are 6.3%, 4.19%, 1.39% respectively, so that the degrees of sparsity are 93.7%, 95.81%, 98.61%.

Quote:

“  we can see the MAE value will gradually increase when the degree of sparsity of the data gradually increases. This shows that the lower the degree of sparsity, more abundant the rating information, more accurate the item similarity computation, eventually leading to smaller error between predicted score and actual score; when the degree of sparsity of the data gradually increases, the speed of reaching the smallest MAE value becomes slower, slower, that is $K$ value becomes bigger. This is because that the higher the degree of sparsity, item similarity computation needs more abundant rating information, resulting in bigger neighborhood set; when the degree of sparsity of the data increases, the rate of decline of MAE becomes faster. “

I have put in bold what I think is not correct, meaning: “when the degree of sparsity of the data gradually increases, the speed of reaching the smallest MAE value becomes slower” and  “when the degree of sparsity of the data increases, the rate of decline of MAE becomes faster.” .

Both phrases either say the same or they are contradicting themselves.


texts: Which scoring for GridSearchCV is best, when imbalanced multiclass dataset?

texts:
I have an unbalanced multiclass dataset (GTSRB) and want to optimize the hyperparameters of an SVM through GridSearchCV.
I know that accuracy is not suitable for scoring in this case.
Which evaluation method for scoring would be most appropriate in this case?

At the moment I tend between the following:
- f1_score (average='macro')
- cohen_kappa_score

What are your experiences in such cases?


texts: Classification problem where one attribute is a vector

texts:
Hello I am a layman trying to analyze game data from League of Legends, specifically looking at predicting the win rate for a given champion given an item build.
Outline
A player can own up to 6 items at the end of a game. They could have purchased these items in different orders or adjusted their inventory position during the course of the game.
In this fashion the dataset may contain the following rows with:
   champion id   |                 items ids               | win(1)/loss(0)
----------------------------------------------------------------------------
       45        |   [3089, 3135, 3151, 3157, 3165, 3285]  |       1
       45        |   [3151, 3285, 3135, 3089, 3157, 3165]  |       1
       45        |   [3165, 3285, 3089, 3135, 3157, 3151]  |       0

While the items are in a different order the build is the same, my initial thought would be to simply multiply the item ids as this would give me an integer value representing that combination of 6 items.
While there are hundreds of items, in reality a champion draws off a small subset (~20) of those to form the core (3 items) of their build. A game may also finish before players have had time to purchase 6 items:
                items ids
------------------------------------------
   [3089, XXXX, 3151, 3285, 3165, 0000]
   [XXXX, 3285, XXXX, 3165, 3151, 0000]
   [3165, 3285, 3089, XXXX, 0000, 0000]

XXXX item from outside core subset
0000 empty inventory slot

As item 3089 compliments champion 45 core builds that have item 3089 have a higher win rate than core builds which are missing item 3089.
The size of the data set available for each champion varies between 10000 and 100000. The mean is probably around 35000.
Questions

Is this a suitable problem for supervised classification?
How should I approach finding groups of core items and their win rates?


texts:
I'm training an LSTM model created with Keras.
During epochs, I notice that the training loss decreases epoch after epoch, but validation loss fluctuates.
The fluctuation is small.

The number of features of my examples are 837.

Model:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=400, return_sequences=True), input_shape=(timesteps, features)))
model.add(Dense(1))
model.add(Activation('relu'))

Losses for the first 10 epochs:

loss: 0.3362 - val_loss: 0.3306
loss: 0.3320 - val_loss: 0.3302
loss: 0.3304 - val_loss: 0.3298
loss: 0.3286 - val_loss: 0.3308
loss: 0.3271 - val_loss: 0.3303
loss: 0.3260 - val_loss: 0.3315
loss: 0.3254 - val_loss: 0.3302
loss: 0.3246 - val_loss: 0.3309
loss: 0.3246 - val_loss: 0.3316
loss: 0.3241 - val_loss: 0.3312

Here's the plot:

Is this a bad behavior?

In general, when it happens?


texts: Neural Network outputing the same value / normalization

texts:
I had a simple neural network that was outputting the same value regardless of the input. During training, it was behaving normally, with training and validation loss diminishing to a floor value.

The data range was in [-1000, +1000].
The model is a 1D convolutional network. If it is useful :

    train_loss = lcategorical_crossentropy(...).mean()

    net = InputLayer((None, net_z, net_x), input_var=input_var)

    net = ConvLayer(net,  16, 9, pad='same' )#, flip_filters=False)
    net = ConvLayer(net,  16, 9, pad='same')#, flip_filters=False)

    net = DenseLayer(net,   num_units=32)
    net = DropoutLayer(net, p=0.5)

    net = DenseLayer(net, num_units=2, nonlinearity=None)
    net = NonlinearityLayer(net, softmax)

This was corrected by normalizing the data in [-1, 1]

What is the reason behind that ? Backpropagation being stuck I guess, but where can I learn about that in depth ?


texts: keras InvalidArgumentError: Incompatible shapes: [10,1] vs. [10,60,60,3600]

texts:
I am trying to train an anime face dataset on an autoencoder.
but at the time of training it gives me this error.
InvalidArgumentError:  Incompatible shapes: [10,1] vs. [10,60,60,3600]

I have a folder with 63,565 color images of anime faces and each image has a different size.
what am I doing wrong?
# importing libraries
!mkdir /content/anime
!mv /content/images /content/anime
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
train_datagen = ImageDataGenerator()

import keras
from keras import layers

# This is the size of our encoded representations
encoding_dim = 32  # 32 floats -> compression of factor 24.5, assuming the input is 784 floats

ancho=60
alto=60
n_pixels=alto*ancho

train_generator = train_datagen.flow_from_directory("/content/anime",
                              target_size =(ancho, alto),
                     batch_size =10)

# This is our input image
input_img = keras.Input(shape=(n_pixels,3))

# "encoded" is the encoded representation of the input
encoded = layers.Dense(encoding_dim, activation='relu')(input_img)
# "decoded" is the lossy reconstruction of the input
decoded = layers.Dense(n_pixels, activation='sigmoid')(encoded)

# This model maps an input to its reconstruction
autoencoder = keras.Model(input_img, decoded)

# This model maps an input to its encoded representation
encoder = keras.Model(input_img, encoded)

# This is our encoded (32-dimensional) input
encoded_input = keras.Input(shape=(encoding_dim,))#la salid del encoder osea la entrada del decoder
# Retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# Create the decoder model
decoder = keras.Model(encoded_input, decoder_layer(encoded_input))

autoencoder.compile(optimizer='adam', loss='mse')
from keras.datasets import mnist
import numpy as np
autoencoder.fit(train_generator,
                epochs=10,
                batch_size=10)

Epoch 1/10
WARNING:tensorflow:Model was constructed with shape (None, 3600, 3) for input Tensor("input_1:0", shape=(None, 3600, 3), dtype=float32), but it was called on an input with incompatible shape (None, None, None, None).
WARNING:tensorflow:Model was constructed with shape (None, 3600, 3) for input Tensor("input_1:0", shape=(None, 3600, 3), dtype=float32), but it was called on an input with incompatible shape (None, None, None, None).
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-13-1b5b29a94954> in <module>()
      5 autoencoder.fit(train_generator,
      6                 epochs=10,
----> 7                 batch_size=10)

8 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

InvalidArgumentError:  Incompatible shapes: [10,1] vs. [10,60,60,3600]
     [[node mean_squared_error/SquaredDifference (defined at <ipython-input-13-1b5b29a94954>:7) ]] [Op:__inference_train_function_2136]

Function call stack:
train_function


texts: Can I scrape data from government websites if there is no mention about commercial usage?

texts: I am trying to be sure that can I scrape government data from several websites if there is no mention of any commercial usage? I am willing to scrape US navy data(Link) and Canada Industrial Data (Link) and not sure if I should do. I personally don't hesitate to scrape govt. data but sometimes it may take the whole company in trouble specially in US. I need some valuable suggestions how to classify any website whether we should scrape or not.


texts:
It has been a long I am confused on understanding some of the AlexNet architecture :
The output of the first conv layer is 55x55x48 (96 considering the division between GPUs but let's stick to 1 GPU so depth 48). Then max pooling is applied and there come my problem.
When applying max pooling, the result is 27x27x48 right ? If so, how is applied the next convolution over this result (with 5x5x48 filters) to output 27x27x128 ? I finally don't see how and when to apply max-pooling in between convolutions. I must miss something here...


texts: predict_proba returns different results on Python 2 & 3

texts:
I had some old code used to train a Random Forest Classifier (sklearn 0.17.1), for classification on two classes (spam/ham). I ran this in a docker container and sent it some data.

Sklearns predict_proba returns [0.3, 0.7].

I “rewrote” (literally changed a few lines for Python3 compatibility), trained with the same data, and sent it the same data, and now predict_proba returns [0.7, 0.3].

Did something change in sklearn.ensemble.RandomForestClassifier predict_proba or have I messed up somewhere, or does the new one still work?


texts: Download weather data from a http directory

texts:
This link contains weather data with a lot of folders and subfolders. I need to download as a whole without visiting every file inside the folders.
https://www.ncei.noaa.gov/data/global-forecast-system/access/historical/analysis/
Any tool or script would help.


texts: LSTM - How to prepare train from a dataset which contains multiple observations for different events

texts:
I m using LSTM in a project related to MobiFall dataset which contains falls and daily activitives - such as walking, sitting etc - sensed by accelerometer, gyroscope and orientation sensors in x,y,z axes.

So I need to modify LSTM into multi-variate form. How could it be done?

And after this problem is solved, I have to deal with another, there are multiple time-series events in different files which were done by different people. For example, I have got ADL_1_walking_1_.txt, ADL_1_walking_2.txt, ADL_2_jumping_3.txt (Naming convention of text files is ActivityType_SubjectNo_ActivityName_SampleNo.txt) which contains acc-x, acc-y,acc-z, gyro-x, gyro-y, gyro-z, ori-x, ori-y, ori-z data where acc means accelerometer, gyro means gyroscope and ori means orientation.

So how should I train my model? Should I combine all of those files? But doing this will cause model to suppose they are all from the same observation right? So it may lead wrong calculations. Should prepare a training and test test for each file and train the model over and over again? So my question is what are the general solutions for this kind of problem. Can you tell me some of these solutions?


texts: Why is my prediction using ARIMA better if I'm using less historic data?

texts:
I have a data set containing hourly electricity prices for since 1.01.19 until September. Since the process turned out to be (weakly) stationary, I applied an ARIMA model in Python in order to predict the prices for the next day.

It turned out that the best prediction was made using the last two days as historic data and the worst was the one using almost 6000 values.

What is a possible explanation for this happening?


texts:
How can I get the feature vector of my dataset. I have a fine-tuned CNN model with my data. Now I want to feed the features of all my dataset extracted from the last layer of the CNN into a LSTM.

So far I have

cnn_model = load_model('weights/pre-trained_CNN.hdf5')
cnn_output = cnn_model.get_layer('fc7').output

I know I will have to feed all my dataset into this model but I have no idea on how to "save" the features.


texts: How can I weigh observations differently that were provided for a time horizon?

texts:
I have 623 observations with one continuous dependent variable and 13 independent variables (continuous, categorical, and ordinal), defined based on researcher experience and literature review.
I considered planning to do several regression analysis to estimate the dependent variable and study the predictive factors (if they are positive, negative, and their magnitude) on it. Data are provided for 10 years. Since the latest observations are more important, I’m interested in using weighted observations. How can I approach this problem and validate my approach?


texts: adding logic combinations of boolean features in classification

texts:
I want to build a classifier from a dataset of vectors that include exclusively boolean values. Is there any chances that my classifier might perform better if, previously to the learning, I add features that consist in combinations (with logic operators « OR » and « AND ») of the original ones or would that be pointless ?

For example I have the following dataset :

            feature A   feature B   feature C
vector 1    True        False       True
vector 2    True        True        False
vector 3    False       True        True

Imagine I have the feeling that the fact that a data has True for both feature C and feature D would typically make it go in one particular class. Before learning the datas to build the classifier, should I add to each vector en extra feature computed by the logical operation « feature B AND feature C » :

            feature A   feature B   feature C   feature D
vector 1    True        False       True        False
vector 2    True        True        False       False
vector 3    False       True        True        True

Or would the relevance of « feature B AND feature C » be taken into account by the classifier anyway ? Does it depend on the algorithm (svm, Knn, etc. ) ?


texts: Exploring variables to guide xgboost tuning

texts:
In short: How to think about the type and distribution of my variables when choosing parameter values for xgboost?
Context: I have a dataset which I want to classify using the binary:logistic objective (I am using the R implementation). I wonder if visually inspecting variables in my dataset could inform selection of parameters values, and if so then how does one generally go about it.
Example: A dataset with a binary outcome variable and a mixture of binary/continuous predictors.
Outcome variable (one value per observation):

A set of predictors called flavours (~150 variables) and textures (~80 variables) have binary values 0, 1. Each observation has one or multiple flavours and textures (number of f/t per observation on the horizontal axis):

Some flavours/textures occur much more likely than others (frequency on horizontal axis):

I added some quantitative variables to summarise flavour/texture for each observation. I calculated a frequency for each f/t (= what is the percentage of observations where this f/t occurs) and then calculated min (most unique f/t in the given observation) max (most common f/t) and mean (average frequency).

Apart from that, there are three more continuous variables:

What conclusions regarding model parameters one might draw from this, if any?


texts: How to change Linear model in SGDClassifier scikit learn?

texts: The SGDClassifier of scikit learn defines it as "Linear classifiers (SVM, logistic regression, etc.) with SGD training.". I understand from this that any Linear classifier can be used here. I want to know that how can we define a specific linear classifier to be used in the case of SGDClassifier.


texts: Methods / Algorithms for rank scales based on cumulative scoring

texts:
Say you have an organization that requires employees to participate in a Q&A site similar to StackOverflow - questions and answers are voted upon, selected answers get extra points, certain behaviors boost your score etc.  What we need to do is assign a rating from 1-100 to these users with even distribution.

The behaviors that add points:

Ask a question [fixed]
Answer a question [fixed]
Receive an upvote on a question [determined by relative ranking]
Receive an upvote on an answer [determined by relative ranking]
Have your answer selected [determined by relative ranking]
Responding to a comment, etc [fixed]

Likewise, there are behaviors that subtract points.

If a user with a high ranking upvotes a question asked by a lower-ranking user, more points should be awarded than the inverse situation.  Likewise if a lower-ranking user downvotes a higher-ranking user's question, the impact should be minimal compared to the inverse.  There should be a limit to this impact though so that a high-ranking user doesn't unintentionally destroy any momentum of a low-ranking user by issuing a powerful downvote.

We have a few challenges here:

How do we determine how many points to assign to each type of behavior, with actor/recipient relative rank taken into account?
I'm thinking we just assign a flat number to each behavior, that number decided relative to the importance of the other behaviors, and then have a variable score that can alter the score if there is a wide variance between the users.  The mechanics of this - does the score double at most? - are unclear.
How to we assign this rank? This one is a little easier - I'm thinking we just order the users according to score and then split the dataset into 100 sections, assigning each "chunk" a number 1-100.
Should we be worried about these numbers getting "very big"?  The scenario described above has been trivialized; actions taken by these users may happen hundreds of times per day so the scores can become very high, very quickly. Is there a way we can keep this under control while avoiding a large number of duplicate scores?
How do we define the "fixed" scores as the total scores become very big?  Over time we may have users with hundreds of thousands of points - but the fixed-score behaviors should still reward them.  They should reward lower-ranking users more than higher-ranking users.

I don't know if there are some standard practices, algorithms, or terminologies that I should be aware of when facing a problem like this - any input would be appreciated.


texts: How Transformer is Bidirectional - Machine Learning

texts:
Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/55158766?noredirect=1#comment97066160_55158766

I am coming from Google BERT context (Bidirectional Encoder representations from Transformers). I have gone through architecture and codes. People say this is bidirectional by nature. To make it unidirectional attention some mask is to be applied.

Basically a transformer takes key, values and queries as input; uses encoder decoder architecture; and applies attention to these keys, queries and values. What I understood is we need to pass tokens explicitly rather than transformer understanding this by nature.

Can someone please explain what makes transformer bidirectional by nature

Answer received so far:
1. People confirmed that Transformer has Bidirectional nature, rather than an external code making it bidirectional.
2.
My doubt: We are passing Q K V embeddings to transformer, to which it applies N layers of self attention using ScaledDotMatrix attention. Same thing can be done by unidirection approach as well. May I know what part I am missing in my understanding. If someone can point to code where it is getting bidirectional, it would be a great help.


texts:
I've little idea about choosing a ML approach for the following problem. It is a classification problem and there are 2 classes that are positive and negative. There are about 100k samples and samples are structured like this:
Period = 1min   Pattern = M1>S1>T1>B2>M2>S2>S3>T3>M3>B3
Period = 5min   Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3
Period = 10min  Pattern = M1>T1>S1>M2>B2>S2>S3>T3>M3>B3
Period = 15min  Pattern = M1>T1>S1>B2>S3>M2>S2>T3>M3>B3
Period = 20min  Pattern = S1>M1>S3>T1>B2>M2>S2>T3>M3>B3
Period = 30min  Pattern = S1>S3>B2>M1>T1>S2>M2>T3>M3>B3
Period = 60min  Pattern = S1>B2>M1>T1>S2>M2>S3>T3>B3>M3
Period = 120min Pattern = S1>M1>T1>B2>S2>M2>T3>S3>M3>B3

This sample is classified as negative. A sample is composed of 8 periods. Within each period there is a pattern such as M1>S1>T1>B2>M2>S2>S3>T3>M3>B3. Each pattern has 10 elements and their positions are changing along samples and periods. We need to come up with a solution that could tell which period or lineup of elements are responsible for classification.
Let's say we have p1, p2, p3 positive examples and n1, n2, n3 negative examples with 1min Periods like this:
p1: M1>S1>T1>B2>M2>S2>S3>T3>M3>B3
p2: M1>S1>T1>B2>S2>M2>S3>T3>M3>B3
p3: M1>S1>T1>B2>M2>S2>S3>T3>M3>B3

n1: M1>S1>T1>B2>S2>M2>S3>T3>B3>M3
n2: M1>S1>T1>B2>M2>S2>S3>T3>B3>M3
n3: M1>S1>T1>B2>M2>S2>S3>T3>B3>M3

It could be inferred that first 4 elements M1,S1,T1, B2 are irrelevant for classification since they are all same across all samples. 5th and 6th elements are also irrelevant since they don't show same pattern along same class of samples. However, elements B3, M3 is a solid positive since M3>B3 for positive samples and B3>M3 for negative samples.
Thanks.


texts:
I am trying to identify what attributes are not relevant in my dataset to remove them before fitting a classifier.

The target is a categorical variable with three different values.

I also have a lot of numerical attributes.

For ANOVA, I used the following code:

grouped_test2=df[['room_type', 'price']].groupby(['room_type'])
f_val, p_val = stats.f_oneway(grouped_test2.get_group('Entire home/apt')['price'], grouped_test2.get_group('Private room')['price'], grouped_test2.get_group('Shared room')['price'])

The independent variable is room_type, and the explanatory variable is price.

In this case, the f_val is equal to 1061.64 and p_val is equal to 0.

I read that 0 or values near 0 imply a relationship between the two variables but I am not sure about that?

What mean f_val is near enough to 0 to can say that the two variables are related?


texts: Tuning the model parameters vs the parameter of optimizer for Deep Neural Networks?

texts:
I understand that there are rarely general recipes in field of machine learning and the many results can be achieved only by trial and error, and are task specific as well.
My question is, if the model doesn't give a desired quality, but one may expect, that the model of this or similar architecture can achieve a reasonable result, based on some prior assumptions or knowledge, what is, in general, better strategy, more successful in majority cases:

Tuning of the optimizer parameters of the optimizer. (playing with the learning rates, random seeds, weight decay, momentum, e.t.c). hoping that it converges to the better optimum
Changing some architecture properties of the model (number of hidden layers, dropout rate, width, number of filters in CNN)

Here I assume, that the current network is sensible choice - not 1-hidden layer narrow MLP - for solving some complicated task.
Minimization of loss function doesn't always lead to improvement of the quality of the model on the test data (well known problem of overfitting), but here let us assume, that one has lot enough data, and the training and test set are organized in a such way, that good score on the training examples would also lead to good accuracy on the test data.


texts: Does autoencoder with few images not work?

texts:
I am trying to load 2 images to an autoencoder but for some reason, it does not rebuild the input image.
An autoencoder is supposed to compress and decompress an image. However, when passing two images with which it trains, it only shows a black color instead of the corresponding image.
import keras
from keras import layers

# This is the size of our encoded representations
encoding_dim = 32  # 32 floats -> compression of factor 24.5, assuming the input is 784 floats

#28*28=784  tamaño de images de nmist

ancho=28
alto=28
n_pixels=alto*ancho

# This is our input image
input_img = keras.Input(shape=(n_pixels,))

# "encoded" is the encoded representation of the input
encoded = layers.Dense(encoding_dim, activation='relu')(input_img)
# "decoded" is the lossy reconstruction of the input
decoded = layers.Dense(n_pixels, activation='sigmoid')(encoded)

# This model maps an input to its reconstruction
autoencoder = keras.Model(input_img, decoded)

# This model maps an input to its encoded representation
encoder = keras.Model(input_img, encoded)

# This is our encoded (32-dimensional) input
encoded_input = keras.Input(shape=(encoding_dim,))#la salid del encoder osea la entrada del decoder
# Retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# Create the decoder model
decoder = keras.Model(encoded_input, decoder_layer(encoded_input))

This code is to load two images with the help of pil module:
##load custom data
from PIL import Image
import numpy as np

#data1=np.linspace(0,0,2)
#data1_test=np.linspace(0,0,2)
data1=[0,0,0]
data1_test=[0,0,0]

data2=np.linspace(0,0,n_pixels*2).reshape(2,n_pixels)

data1[0]=Image.open("/content/zero.png")
data1[1]=Image.open("/content/eight.png")

#custom_imagen_arr=np.array(list(custom_imagen)[0]).reshape(1,784)
custom_data_arr=np.linspace(0,0,n_pixels*2).reshape(2,n_pixels)
for  i2 in range(2):
       imm=list(data1[i2].getdata())
       for  i in range(len(list(data1[i2].getdata()))):
            custom_data_arr[1,i]=imm[i][0] / 255.
       for  i in range(len(list(data1[i2].getdata()))):
            data2[i2][i]=custom_data_arr[1,i]
data2 = data2.astype('float32') / 255.

data2 = data2.reshape((len(data2), np.prod(data2.shape[1:])))

Then I train:
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
import numpy as np
autoencoder.fit(data2, data2,
                epochs=152,
                batch_size=1,
                verbose=1,
                validation_data=(data2, data2))

After finishing training, I choose one of the loaded images and make a prediction, but the output is black.
#give an arbitrary image to the autoencoder and generate an image

from PIL import Image
custom_imagen=Image.open("/content/eight.png")
#custom_imagen_arr=np.array(list(custom_imagen)[0]).reshape(1,784)
custom_imagen_arr=np.linspace(1,0,n_pixels*2).reshape(2,n_pixels)
imm=list(custom_imagen.getdata())
for  i in range(len(list(custom_imagen.getdata()))):
        custom_imagen_arr[1,i]=imm[i][0] / 255.
encoded_imgs = encoder.predict(custom_imagen_arr)
decoded_imgs = decoder.predict(encoded_imgs)

# Use Matplotlib (don't ask)
import matplotlib.pyplot as plt

n = 2  # How many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
    # Display original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(custom_imagen_arr[i].reshape(alto, ancho))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # Display reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(alto, ancho))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()

How do I so that the output of the autoencoder is not black and the same number is the one I train?
The code is modified from here.


texts: R random forest on Amazon ec2 Error: cannot allocate vector of size 5.4 Gb

texts:
I am training random forest models in R using randomForest() with 1000 trees and data frames with about 20 predictors and 600K rows. On my laptop everything works fine, but when I move to amazon ec2, to run the same thing, I get the error:

Error: cannot allocate vector of size 5.4 Gb
Execution halted

I'm using the c3.4xlarge instance type so it's pretty beefy. Does anyone know a workaround for this to get it to run on this instance? I would love to know the memory nuances that causes this problem only on the ec2 instance and not on my laptop (OS X 10.9.5 Processor  2.7 GHz Intel Core i7; Memory  16 GB 1600 MHz DDR3)

Thanks.


texts:
Details:
GPU: GTX 1080
Training: ~1.1 Million images belonging to 10 classes
Validation: ~150 Thousand images belonging to 10 classes
Time per Epoch: ~10 hours
I've setup CUDA, cuDNN and Tensorflow( Tensorflow GPU as well).
I don't think my model is that complicated that is takes 10 hours per epoch. I even checked if my GPU was the problem but it wasn't.
Is the training time due to the Fully connected layers?
My model:
model = Sequential()
model.add()
model.add(Conv2D(64, (3, 3), padding="same", strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding="same", strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=2))
model.add(Flatten())
model.add(Dense(256))
model.add(Activation('relu'))
model.add(Dense(4096))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.summary()

opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)

model.compile(loss='categorical_crossentropy',
          optimizer=opt,
          metrics=['accuracy']
          )

Because there is a lot of data I used the ImageDataGenerator.
gen = ImageDataGenerator(
 horizontal_flip=True
)

train_gen = gen.flow_from_directory(
        'train/',
        target_size=(512, 512),
        batch_size=5,
        class_mode="categorical"
)

valid_gen = gen.flow_from_directory(
        'validation/',
        target_size=(512, 512),
        batch_size=5,
        class_mode="categorical"
)


texts: faster alternatives to sparse.model.matrix?

texts:
I have a large dataset that is entirely categorical. I'm trying to train with it using xgboost, so I must first convert this categorical data to numerical. So far I've been using sparse.model.matrix() in the Matrix library but it is far too slow. I found a great solution here, however, the sparse matrix it returns in not the same one that sparse.model.matrix returns. I know there is a way to force sparse.model.matrix to return identical output as the solution in the link (by providing contrasts), however, that is not an effective solution as it is still too slow and results in different representation (and hence different training model).

Is there a way to accomplish the job sparse.model.matrix does even nearly as fast as the solution I posted? For my data, the solution I posted does it in about 15% of the time.


texts: How to efficiently iterate a supervised model over the Cartesian product of very large number of records?

texts:
The problem:

Two large databases, with ~1M records each, "old customer data" and "new customer data". The data came from different sources and was ingested at different times, so there are many duplicates, but the duplicates might not be exact matches. For example In the old data, the customer was listed as "Michael Smith" but in the new data, they were listed as "Mike Smith" or "M. Smith", or the names match, but the address field is different: How do we now if it is different people with the same name or the same person who changed address?

Approach:

It seems possible to use a supervised learning approach to classify a pair of records as either being duplicates or not.

Question:

Assuming such a model is possible and we have trained it, how would we iterate over the entire data set to produce our predictions?

To apply a supervised approach (or any ML/Probabilistic approach), I can only think of naively examining each pair of records one-by-one, but this means our models would have to iterate through $10^{12}$ records, which doesn't seem feasible even with advanced compute capabilities?

How can one efficiently iterate/search through the combined data set in a situation like this?


texts: XGBoost is it possible to prevent a feature from being used twice in the same tree?

texts:
I'm using XGBoost and all its doing is using the feature in the first column of my data. My feature importance chart correlates perfectly to the position of the feature in my xtrain. If I shuffle the columns in xtrain, the feature chart changes along witht he shuffle after re-running the model.
This suggests that XGBoost is "getting stuck" on the first feature. Is there a way to prevent this? I was thinking by limiting how many times the same feature can appear in a tree?


texts: A math question about solving the Lagrangian of Support Vector Machine

texts:
$$\mathcal{L}(w,b,\xi,\alpha,r) = \frac12w^Tw+C\sum_{i=1}^m \xi_i-\sum_{i=1}^m \alpha_i[y^{(i)}(x^Tw+b)-1+\xi_i]-\sum_{i=1}^mr_i\xi_i$$

Here, the $\alpha_i$'s and $r_i$'s are our Lagrange multipliers (constrained to be $\ge 0$)

To maximize the Lagrangian of soft margin SVM (see the formula above), we set the derivatives with respect to $w$, $\xi$ and $b$ to $0$ respectively.

But what if we set the derivatives w.r.t $r$ to zero first? Wouldn't that result in $\xi$ being all $0$s? Meaning that the optimal solution is reached only when all the relaxing terms $\xi$ are $0$? But that doesn't seem right, does it?


texts: How to estimate Distance to Obstacles using Lidar's BEV(Bir's Eye view ) Representation?

texts: I am using a implementation that use both camera and Lidar for 3D obstacle detection for self driving cars but I have no idea how to calculate the distance to the obstacles since the Lidar uses the Bird's Eye View ?


texts: Decreasing n_estimators is increasing accuracy in AdaBoost?

texts:
I was exploring the AdaBoost classifier in sklearn. This is the plot of the dataset.
(X,Y are the predictor columns and the color is the label)

As you can see there are exactly 16 points in either side that can be easily miss-classified. TO check how the performance increases with increasing n_estimators I used this code

for i in range(1,21):
    clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2),
            n_estimators=i,algorithm='SAMME')
    clf.fit(X_df,y)
    y_pred = clf.predict(X_df)

    from sklearn.metrics import confusion_matrix as CM
    #CM(y_pred,y_pred1)
    #CM(y,y_pred1)
    print(i,CM(y,y_pred))

Upto n_estimators = 13 all the 32 points were miss classified. The confusion matrix is

[[84 16]
 [16 84]]

(
with the exception of n_estimators=8. Here all red points were properly classified)

[[100 0]
 [16 84]]

From 13 onwards it began flipping in a weird way. The confusion matrices are
 given in order.

[[84 16] | [[ 84  16] | [[84 16] | [[ 84  16] | [[ 84  16] | [[100   0] | [[100   0] | [[84 16]
[16 84]] | [  0 100]] | [16 84]] | [  0 100]] | [  0 100]] | [ 16  84]] | [  0 100]] | [16 84]]

Where apparently n_estimators=19 is giving better performance than n_estimators = 20.

Can someone please explain what is happening and what is causing this behavior?


texts: Verifying the implementation of Multihead Attention in Transformer

texts:
I have implemented the MultiAttention head in Transformers. There are so many implementations around so its confusing. Can someone please verify if my implementation is correct:
import tensorflow as tf

def scaled_dot_product(q,k,v):

    #calculates Q . K(transpose)
    qkt = tf.matmul(q,k,transpose_b=True)
    #caculates scaling factor
    dk = tf.math.sqrt(tf.cast(q.shape[-1],dtype=tf.float32))
    scaled_qkt = qkt/dk
    softmax = tf.nn.softmax(scaled_qkt,axis=-1)

    z = tf.matmul(softmax,v)
    #shape: (m,Tx,depth), same shape as q,k,v
    return z

class MultiAttention(tf.keras.layers.Layer):

    def __init__(self,d_model,num_of_heads):

        super(MultiAttention,self).__init__()
        self.d_model = d_model
        self.num_of_heads = num_of_heads
        self.depth = d_model//num_of_heads
        self.wq = [tf.keras.layers.Dense(self.depth) for i in range(num_of_heads)]
        self.wk = [tf.keras.layers.Dense(self.depth) for i in range(num_of_heads)]
        self.wv = [tf.keras.layers.Dense(self.depth) for i in range(num_of_heads)]
        self.wo = tf.keras.layers.Dense(d_model)

    def call(self,x):

        multi_attn = []
        for i in range(self.num_of_heads):
            Q = self.wq[i](x)
            K = self.wk[i](x)
            V = self.wv[i](x)
            multi_attn.append(scaled_dot_product(Q,K,V))

        multi_head = tf.concat(multi_attn,axis=-1)
        multi_head_attention = self.wo(multi_head)
        return multi_head_attention

#Calling the attention
multi = MultiAttention(d_model=512,num_of_heads=8)
m = 5; sequence_length = 4; word_embedding_dim = 512
sample_ip = tf.constant(tf.random.normal(shape=(m,sequence_length,word_embedding_dim)))
attn =multi(sample_ip)
#shape of op (attn): (5,4,512)


texts: How to treat input that inherently has a tree structure?

texts:
When having a single vector you use an MLP neural network
When having a 2D structure you use a CNN neural network
When having a sequence you use a RNN neural network

Now you have preprocessed an instance and the result is a tree structure.
Let's say for simplicity that the tree structure is always the same tree; only the node values differ among instances.

What kind of neural network architecture would be required to consume the information of a tree structure but also leveraging the connections between the tree nodes?


texts: how to do topic modeling on very huge data?

texts: When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every token is important so i would have huge amount of token at that time gensim algo like(LDA,LSA,etc) fails and get out of memory execution issue. Actually they written that there is no limit for amount of document but internally they keep the dictionary size fixed that's why if new word will come and exceed the limit than it starts to truncate it. I dont want to truncate my token as possible. So is there any solution for that can resolve my issue of topic modeling on big data. I have resource limitation.


texts:
I am dealing with the below data:
\begin{array} {|c|c|}\hline \text{user} & \text{A_PDE} & \text{B_PDE} & \text{C_PDE} & \text{D_PDE} & \text{Sales} & y_\text{pred} & \text{SSE} & \hat{y} \\ \hline 1 & 0.50 & 0.25 & 0 & 1.75 & 0 & 8.660412 & 75.00272 & 0 \\ \hline 2 & 0.00 & 0.00 & 1 & 0.00 & 0 & 4.02256 & 16.18099 & 0 \\ \hline 3 & 0.50 & 1.25 & 1 & 0.75 & 44 & 13.99656 & 900.2064 & 615.8487 \\ \hline 4 & 1.25 & 1.00 & 0 & 0.00 & 0 & 11.02223 & 121.4896 & 0 \\ \hline 5 & 0.00 & 0.75 & 0 & 1.50 & 0 & 7.240974 & 52.43171 & 0 \\ \hline  \end{array}

I am trying to calculate the impact of each of the channels A_PDE, B_PDE, C_PDE, D_PDE. However, since Sales = 0 for most of the records, and my model is predicting non zero sales for those records, my $R^2$ value is very low. How can I fix this? I don't think removing the zero sales record from the model would be right.


texts: Multiple Instance Learning when different number of instances in each bag

texts:
I am trying to do a Multiple Instance Learning for a binary classification problem, where each bag of instances has an associated label 0/1. However, the different bags have different numbers of instances. One solution is to take the minimum of all the instance numbers of the bag. For eg-

Bag1 - 20 instances, Bag2- 5 instances, Bag3 - 10 instances .... etc

I am taking the minimum i.e- 5 instances from all the bags. However, this technique discards all the other instances from other bags which might contribute to the training.
Is there any workaround/algorithm for MIL where variable instance numbers for bags could be handled?


texts: Keras decision threshold for Multiple Label prediction

texts:
I'm training a Neural Network to predict multiple labels for a given input. My input is a 200 sized vector of integers and the output should be a boolean vector of size 28. My y has a 1 on the corresponding classes the example corresponds to, i.e. the y should look like:
[0, 0, 1, 0, ... 1, 0, 0].

Now, I've used a sigmoid function on my final layer, and I'm training with binary crossentropy since I want my model to treat each class as independent from each other.

When I fit my model, I get a pretty decent categorical accuracy, i.e. ~75%, but I'm wondering which is the threshold that Keras uses to say a class is or isn't present on the example, I mean, which is the threshold where it decides:

prediction[prediction >= threshold] = 1
prediction[prediction < threshold] = 0


texts: Orange Biolab custom widget learner input

texts:
I was able to install the demo widget in Orange 3.9.1 and made it work with these learners

Naive bayes
Logistic regression

In other words I was able to connect the nodes to the newly installed "Learning curve A" widget.

This is the catch: Those two were the only I was able to connect.
For example, I can't connect the tree learner or the random forest learnear.

You can see what I mean from the picture below.
Find here the source of the demo widget
In the tutorial you can see other learners are connected successfully.

Any idea why I can't connect the other learners?


texts: How do I include the Bias term in the Pegasos algorithm?

texts:
I have been asked to implement the Pegasos algorithm as below.  It is similar to the Peceptron algorithm but includes eta and lambda terms.  However, there is no bias term below and I don't know how to include it in either the condition or the update.  I think the update is just W0 -> W0 + eta*y[i,t], but it should only be updated if the condition is satisfied, and I used the condition below which ignores bias.

Any ideas how to include the bias in the condition and update?


texts: Biasing SVM algorithm towards particular subset of data

texts:
I'm training an SVM model for sentiment analysis, based on social media data eg. tweets.
The model will be trained using a small selection of a particular company's tweets in order to classify new ones. However, since the training set is too small to get an accurate model I will be combining the company's data with a much larger general tweets dataset to train the model.
Being specialised to one company, the content of the respective data is slightly different to the content of the general dataset. Since the data to be predicted is company specialised, it seems logical to me to bias the models training  towards giving greater importance to the company related tweets to improve the accuracy. My first thought was simply increasing the magnitude of the polarity of the companies tweets, ie general tweets are -1 or 1 and company tweets are -3 and 3, for example.
Is this the right idea/method?


texts: Does it make sense to use a learning rate scheduler with an adaptive learning rate method?

texts:
I'm training a large variety of networks and found out that, while using Adadelta as an optimizer, applying also the keras callback ReduceLRonPlateau also helps improve the results of my network.
What I'm concerned is in the fact that Adadelta modifies the learning rate, but also does the ReduceLRonPlateau, but after making this weird fusion the LR is kept at maximum until ReduceLRonPlateau activates, so I don't know if Adadelta is changing it, or maybe ReduceLRonPlateau is blocking the value.
In any case, using both the optimizer and the ReduceLRonPlateau is a good or bad idea? Does it affect much the outcome of the training process?


texts: What is the appropriate approach for training time series data against multiple, consecutive labels?

texts:
Let's say we have a time series $\{{\bf x}_i\}$ of features and are trying to learn to predict a time series $\{t_i\}$ using a neural network. Our goal is to be able to predict the time series value $t$ for both tomorrow and the day after tomorrow given the features ${\bf x}$ observed today.

My initial thoughts are:

If our goal is to be able to predict $t$ tomorrow given features ${\bf x}$ observed today, we would make our (training, labels) dataset $({\bf x}_i, t_{i+1})$.
If our goal is to be able to predict $t$ the day after tomorrow given features ${\bf x}$ observed today, we would make our (training, labels) dataset $({\bf x}_i, t_{i+2})$.

We could of course use two separate NNs to do this with a single output node but I think it would be better to use one with 2 output nodes and train on pairs $(t_{i+1}, t_{i+2})$.

Anyhow, ideally, the model should wind up predicting the same value for $t_{i+2}$ given ${\bf x}_{i+1}$ (when day $i+2$ is tomorrow) as it does when given ${\bf x}_{i}$ (when the day $i+2$ is the day after tomorrow). This is different from trying to predict two independent values with no time translation property. Generally when data has additional structure, it can be exploited in some way.

Can this desired equivalence be built into the training or architecture of the model?


texts: Considerations for multiple hidden layers in neural networks?

texts:
What is a general approach or planning when it comes to selecting the number of hidden layers in a neural network? Are there datasets/industries/classification v. linear that would help in choosing the number of hidden layers?

I mostly deal with small, financial datasets (less than 10,000 records) and want to put more planning into the number of layers that I use. What should I be taking into account?