texts: Url string processing: what is the best way?

texts:
I have ~1000 different news websites and I scraped and saved all the internal url links for each website. For instance, the website dcgazette.com has a 2MB text file with associated urls:

1) https://dcgazette.com/writers-wanted/

2) https://dcgazette.com/2019/fifteen-things-that-caught-my-eye-today-june-20-2019/

3) https://dcgazette.com/page/494/

4) https://dcgazette.com/page/3/

5) https://dcgazette.com/2019/trump-has-reason-to-be-concerned-about-2020/

etc

Notice that urls 2 and 5 are associated with actual articles. Across websites, each has a different way of representing their article urls. But within each website, all article urls have the same format, i.e. "domain/date/title" (for dcgazette.com).

Is there any unsupervised learning algorithm to process these url strings such that I can cluster all article urls together for each domain? The result I am aiming for is extracting all article urls for each domain with as little noise as possibe from non-article urls for all these domains.

An idea I had was separating each url into pieces by splitting on "/" and generating features such as length or type of letters used for each piece and using a k-means clustering algorithm. I'm not sure how this will actually perform however and am open to advice on any methodology that can make this as accurate as possible


texts: overfitting of regression tree from the beginning

texts:
I am trying to build a regression tree to model insurance claim frequencies.
I have 36000 observations and 9 covariates.

My model overfits right from the beginning!, ie as the cost complexity goes down,
the relative validation error goes up.
I am using this command in R:

rpart_model <-rpart(cbind(Duration,Nbclaims) ~ Gender + DriverAge +
  CarAge + Area + Leasing + Power + Fract + Contract, data = motor,    method="poisson",parms=list(shrink=1), control=rpart.control(cp=0))

and obtain these results:

        CP nsplit rel error xerror     xstd
1   1.9019e-03        0     1.00000   1.0002   0.022911

2   1.3854e-03        1     0.99810   1.0004   0.022973

3   1.1587e-03        2     0.99671   1.0057   0.023185

4   9.1009e-04        5     0.99324   1.0069   0.023269

5   9.0852e-04        6     0.99233   1.0134   0.023526

6   8.8781e-04        7     0.99142   1.0134   0.023531

7   8.5397e-04       10     0.98872   1.0150   0.023603

8   8.5119e-04       11     0.98786   1.0153   0.023621

I have done my research to try and figure this issue out but am completely stuck for the moment, could anyone help with this?


texts: Parallel inference for 1000 different Tensorflow models using the GPU?

texts:
I have a 1000 Tensorflow models with the same architecture but having different weights each. For each of these models I would like to do inference, whereas each of the models also receives different inputs (of the same size). This task seemed to me like something that would benefit from the parallelization capabilities of the GPU. Although that is the case I did not find any documentation on doing this kind of parallel inference. Is anyone aware of a technique that would speed up my inference? Perhaps using the GPU?
The most basic form of what I am trying to achieve is this.
dataset = [np.random.rand(100,100) for i in range(1000)]
models = []
for _ in range(1000):
    input_layer = Input(shape=(100,100))
    layer1 = Dense(512)(input_layer)
    output = Dense(1)(layer1)
    models.append(Model(inputs=input_layer, outputs=output))

for i, model in enumerate(models):
    y = model.predict[dataset[i]]

I though it would be beneficial to put all 1000 models into a single big Tensorflow model (see code below). Although this resulted in a speedup, it was only 20%. While GPU utilization is still at 1%.
dataset = [np.random.rand(100,100) for i in range(1000)]
multiple_inputs = []
multiple_outputs = []
for i in range(1000):
    input = Input(shape=(100,100))
    layer1 = Dense(512)(input)
    output = Dense(1)(layer1)
    multiple_inputs.append(input)
    multiple_outputs.append(output)
model = Model(inputs=multiple_inputs, outputs=multiple_outputs)
model.predict(dataset)

Is there a better way do improve the performance?


texts: How to architect ConvNet to ignore top half of image

texts:
I'm building a convoluted neural network to teach a toy car, powered by a Raspberry Pi, how to drive based on incoming streams of frames from a webcam mounted on top of the car. The top half of each image is irrelevant. What matters is the curvature of the road, and this is in the bottom half.

I've generated a substantial amount of data (about 40k records) by driving the car around myself and recording what I do (commands are left, right, and straight) and what the frames are. However my trained ConvNets aren't giving me the performance I'd hoped for. My experimentation with one of the deployed models confirms that it is indeed tricked by changes in the top half of the streaming images.

A simple solution is to programmatically cut the frames in half so that the neural net only receives relevant portions. However, deep neural nets are praised for their ability to learn features with (ideally) zero human future transformations, so I want to avoid this approach. This project is meant to be a learning experience for me so I want learn how to architect the ConvNet more effectively. Each training session takes several hours to run, so rather than try everything under the sun, I'd figure I'd reach out to the community here to narrow my focus of exploration.

One thought I have is to put a fully connected (FC) layer in front so that subsequent layers effectively only convolve over the relevant portion of the image. I think this could potentially work if this up front FC layer learned to assign very small weights to pixels in the top half of the image. Could this work? Are there better architectures?


texts: Method of finding threshold in Decision tree for continuous data

texts: I am using decision tree in Weka and I have some continuous data, so when I use Weka it automatically find the threshold for me but for some reason I want to implement Decision Tree by myself so I need to know what approach to use to find the threshold to discretize my continuous data?


texts: How to perform SoftMax on outputs from multiple models?

texts:
So assuming I wish to classify 4 classes.
Instead of building a 4-class classifier, I build 4 binary classifiers. Each binary classifier differentiates one class from the other three classes.
Now, I have done the training for the 4 classifiers and saved it to local.
I can load the four models using tensorflow.models.load_model.
My question is :
How can I run a SoftMax function to identify which classifier has predicted 'class 0' with the highest confidence?
I can write a loop to check the 4 predictions and max() to find the highest one. But this is giving me trouble as I cannot use the results to perform confusion_matrix  and classification_report from sklearn.metrics.
Can anyone help with codes?


texts: Binary classification with unexplained data

texts:
My apologies for cross-posting to stackoverflow and cross validated. Not really sure which one is the most relevant place.

Please shed some light on me with this task.

Description

Assuming the training data looks like

0 | X773579,Y2640,Y2072,Z4,Z15
1 | X374616379,X773579,X344420902,Y1940,Y1705,Z4,Z15,Z26
...

One would like to predict 0/1 from those comma separated values, without further explanation about what exactly the X, Y, and Z with numbers mean. Some rare cases even have un-numbered X or Y (possibly just noises though). The length of comma separated values varies, but each row contains no repeated values. Also, the length of numbers seem correlated to X, Y, and Z.

The data size is 1 million rows.

Feature

In LibSVM format

Label | Value_ID:1...

Example

0 1:1 2:1 3:1 4:1 5:1
1 1:1 4:1 5:1 14:1 36:1 37:1 38:1 39:1

I've tried treating those values categorically and applying several different models of Spark MLlib, namely linear regression, logistic regression, decision tree, random forests, gradient-boosted trees, and SVM, along with LibSVM and Conditional Random Fields. Their 5-fold cross-validations got only 75%~79% accuracy.

I really want to know if there's any formal way to discover the signal from the data, build up models, extract and even select features accordingly. Admittedly I haven't done ANOVA yet.

Thank you!

Background (self-disclosure)

This is a private interview question and the employer didn't share any details (except "no offer" response :p). Although I specifically asked twice that I would like to learn from it, no reply so far.

Book studying/studied

Pattern classification
Pattern recognition
Foundations of Statistical Natural Language Processing
Speech and Language Processing

What I may try next

Based on a discussion with @NeilSlater :

I have done some quick frequent pattern analysis and found certain n-grams. The problem was that they seemed associated with both labels 0 and 1. I will run some feature templates of bi-gram and tri-gram, and then get back to you. One thing that is also intriguing to me is, in NLP, sometimes labels also got dependencies, namely some 1 may be followed by certain patterns of 0 and vice versa.

Updates

In terms of linear/logistic regressions and conditional random fields:

n-gram is slightly harmful: tried adjacent bigram/trigram & pair/triple combinations, all got worse about 1%
occurrences of X/Y/Z & mean/max/min/sum/avg of X/Y/Z can only reach 69%.
X or Y or Z solely as categorical feature got 70%~75%. Y is the most useful category.

Other models listed in the original description all got far worse results.


texts: Do I need to use learning algorithms for this type of data?

texts:
I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or not there was a failure in each subperiod. The failures were generated according to a exponential distribution. I have 100 features (Period_1 to Period_100), each containing information for 112 subperiods. My intention is to predict the behavior of the failures for the next period. However, I have two questions:

Is a learning algorithm really necessary to predict the next period failures or could some simpler statistical method be used?
The 100 features represent the same type of information, but for different periods. Would I have a problem with that?

Any help will be appreciated.
A sample of the data:
    Period_1 Period_2 Period_3 Period_4 Period_5 Period_6 Period_7 Period_8
1     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
2     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
3     Normal   Normal  Failure   Normal   Normal   Normal   Normal   Normal
4     Normal  Failure   Normal   Normal   Normal   Normal   Normal  Failure
5     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
6     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
7     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
8     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
9     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
10    Normal  Failure   Normal   Normal   Normal   Normal   Normal   Normal

In this moment, I'm using several classification methods to predict the failures: gradient boosting algorithm, random forest, Classification and Regression Trees, neural networks, Bagged CART, SVM, C5.0, eXtreme Gradient Boosting, and k-Nearest Neighbors. For all this, I use strategy to deal with the imbalance.


texts:
I'm working on a classification problem .I want to classify Iris flowers from the famous Iris data set using MLP. I know that I the number of neurons in output layer should be the same number of classes but can I use one neuron in output layer which output is the value (0 or 1 or -1) to refer to the three types or then it is considered as regression not classification ???? thanks

trin= [4.7 3.2 1.6 0.2;
   4.8 3.1 1.6 0.2;
   5.4 3.4 1.5 0.4;
   5.2 4.1 1.5 0.1;
   5.5 4.2 1.4 0.2;
  5.7 2.6 3.5 1;
  5.5 2.4 3.8 1.1;
  5.5 2.4 3.7 1;
  5.8 2.7 3.9 1.2;
  6 2.7 5.1 1.6;
  6.7 3.3 5.7 2.1;
  7.2 3.2 6 1.8;
  6.2 2.8 4.8 1.8;
  6.1 3 4.9 1.8;
  6.4 2.8 5.6 2.1
  ];
   trout=[-1;-1;-1;-1;-1;
          0;0;0;0;0;
         1;1;1;1;1];
         inp=size(trin,2);
         out=size(trout,2);
          hidden=2;

      x=[-0.8000,-1.520,-0.9400,-3.040,3.800,2,-2,3.790,-1,0,4.600,4.400,0];
 iw = reshape(x(1:hidden*inp),hidden,inp);
 b1 = reshape(x(hidden*inp+1:hidden*inp+hidden),hidden,1);
 lw = reshape(x(hidden*inp+hidden+1:hidden*inp+hidden+hidden*out),out,hidden);
 b2 = reshape(x(hidden*inp+hidden+hidden*out+1:hidden*inp+hidden+hidden*out+out),out,1);

 y =

tanh(tanh(trin*iw'+repmat(b1',size(trin,1),1))*lw'+repmat(b2',size(trin,1),1));
    e = gsubtract(trout,y);

is this classification or it is considered as regression . I mean should I make the out put 3 bits to be consedered as classification and how to do this if yes?


texts:
I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK

But it seems spark do not have this algorithm any more and some forums suggesting me to use ALS, but please clarify is item similarity removed from spark or any other alter natives?
 I want to index my indicators into search engine and I find Sparks Item similarity best suits for me. Please advice.


texts:
I am trying to build a liveness detection system but till now I haven't been able to generate any significant results.
I read the article by pyimagesearch this on the same topic but no luck. Tried using some code from the already existing repos but no results.
I would like to know is there a particular approach if so, it'd be appreciated if you can explain it and if possible provide references, code, databases, etc. also.
Thanks in advance


texts: Entropy loss from collapsing/merging two categories

texts:
Suppose I am counting occurrences in a sequence. For a classical example, let's say I'm counting how many of each kind of car comes down a highway.

After keeping tally for a while, I see there are thousands of models. But only a handful show up frequently, whereas there are many that show up once or only a few times (iow the histogram resembles exponential decay). When thinking about the statistics of this situation, it hardly seems to matter that I saw this one obscure car this one time as opposed to another obscure car - it doesn't seem like it conveys much information either way. So what if I collapse all the rare models into a single category like "others", to make the data easier to store. How much information will I lose?

I've gotten as far as reducing the problem to a smaller one and finding an upper bound.

Collapsing 3 categories A, B and C into one category D is the same as first collapsing categories A and B into category E, then collapsing E and C into F. F will be exactly the same as D. So the final information loss is not path-dependent, and it is sufficient for us to solve information loss from collapsing 2 categories. The result should easily generalize to n categories.
For 2 categories, we can re-encode the sequence so that each occurrence of categories A and B is instead recorded as C. However, for each occurrence of C, an additional bit is recorded that shows whether that C came about from A or B. This re-encoding incurs zero information loss. Erasing these bits would effectively collapse A and B into C. Therefore the average information loss from collapsing categories A and B is (1 bit) * ((number of occurrences of A) + (number of occurrences of B)).

Is my logic above correct? Is my upper bound accurate? What is the lower bound/exact solution?


texts:
I am doing experiments on the high-dimensional regression. However, it is hard to obtain the practical or synthetic high-dimensional data. I have checked on UCI website as well as some papers with related to this topic. Currently, I have not obtained satisfactory datasets.

Could someone give me some hints on how to search and obtain the high-dimensional data? Thanks in advance.


texts: Pretrained word embeddings vs model weights

texts:
I'm trying to understand the relationship between pretrained word vectors and pretrained  weights when using pretrained word vectors in another neural network.

Are the vectors themselves the weights or do the weights need to be transferred in addition to selecting embedded representations? What's the high level conceptual process in using pretrained vectors?

Thanks!


texts:
Can someone provide a summary of the real world deployment experience of MLflow? We have a few ML models (e.g., LightGBM, tensorflow v2, etc.) and want to avoid framework like SageMaker (due to customer requirement). So we are looking into various ways of hosting ML models for inferencing. Latency is one key performance metric that is very important to us. MLflow looks like a good choice for us.
It would be greatly appreciated if the users of MLflow can share their experience.


texts: Binary classification or Single-class classification for data with Boolean label

texts:
In supervised learning. For simple prediction/classification problems like Will it rain tomorrow? or more serious one like disease diagnosis
I often encountering data that has lots of features(images, sensor readings, etc.) on the X side, but only a boolean true or false on the Y side.
Which is better?

Single-class classification that encodes true to 1.0 and false to 0.0 or -1.0
Binary classification that has two classes where we encode the label to each class like true to [0, 1] or [-1, 1] and false to [1,0] or [1,-1]

(which one is better to encode false to 0 or -1? In one-hot we use 0 but some other times we use -1?)
I always stumbled on this problem when I start a new project.
My first instinct goes because it's a clear true and false problem, let's have a single-class approach.
But I kept finding that someone did it with better outcome through encoding the binary labels to logits.
Or somehow tf.nn.sigmoid_cross_entropy_with_logits performed better(or sometimes worse) compared to their non-logits counter part.
I found it confusing that frameworks like PyTorch provides both logits and non-logits version for the same thing. (torch.nn.BCELoss vs torch.nn.BCEWithLogitsLoss)
What are the rules(of thumb) to follow?


texts:
Are there any rules of thumb (or actual rules) pertaining to the minimum, maximum and "reasonable" amount of LSTM cells I should use? Specifically I am relating to BasicLSTMCell from TensorFlow and num_units property.

Please assume that I have a classification problem defined by:

t - number of time steps
n - length of input vector in each time step
m - length of output vector (number of classes)
i - number of training examples

Is it true, for example, that the number of training examples should be larger than:

4*((n+1)*m + m*m)*c

where c is number of cells? I based this on this: How to calculate the number of parameters of an LSTM network? As I understand, this should give the total number of parameters, which should be less than number of training examples.


texts: Importance of Random initialisation VS number of hidden units

texts:
A question crossed my mind not so long ago: I am doing experiments on Language Model with RNN (always with the same network topology: 50 hidden units, and 10M "directs connections" that are emulating N_grams models) and different fraction of corpus (10,25,50,75,100%) (9M words).
I noticed that while perplexity seems to decrease when the training data become more abundant, certain times it does not.
Last example : 143 118 109 106 112
My first thought was network initialization, so I began testing with a smaller corpus and 20 hidden units (for technical reasons. Even with 10% corpus, learning can take up to 30h, which is problematic for me), and I found after 50 tries that all nets converged on values within 3% of each other.
But, I thought that maybe the importance of this initialization is a function of the number of hidden units? I mean the more hidden units the more parameters to tune.
Also, maybe my stop criterion is too sensitive (It stops if evolution of perplexity between two iterations is inferior to a certain number).
Do you think it would make an impact to allow it to run one of two iterations after the criterion was met to see if it was just a local thing?


texts: Does this notation mean vector-concatenation?

texts:
When reading papers on neural networks, I occasionally stumble upon the following notation with a semicolon:

$$
\text{tanh}(\mathbf{W_c}[\mathbf{c}_t;\mathbf{h}_t])
$$

Unless otherwise noted, does this by default mean the following:

vector $\mathbf{c}_t$ is appended to vector $\mathbf{h}_t$
the resulting long vector is dot-producted with a Weight matrix $\mathbf{W}_t$
Finally, the resulting vector is component-wise activated by a hyperbolic tangent function

The first bullet point is my main question.
Googling for "Vector concatenation notation" doesn't return answers that would resemble the image I've attached above. However, many papers seem to use it.

An example of notation that I googled


texts: Multiple activation functions with TensorFlow estimator DNNClassifier

texts:
I just want to know if is it possible to use tf.estimator.DNNClassifier with multiple different activation functions. I mean, could I use a DNNClassifier estimator which use different activation functions for different layers?

For example, if I have a three layers model, could I use for the first layer a sigmoid function, for the second one a ReLu function and finally for the last one a tanh function?

I would like to know if it isn't possible to do it with DNNClassifier how can I do it by a easy way.


texts: How do I find the correct decay rate when the data are not helping?

texts:
If I am making nonsense, I beg some mercy. I am only 17 and is not exactly a college freshmen.

In Nate's Silvers methodology of predicting election result, he explained that each survey has a decay rate of 200 days. That means, if a new data weights  full, data that was published 200 days ago only weights the half of the new day.

I mean, it makes senses. A poll of Hillary v Trump D-5 of election is more accurate compared to Hillary v Trump D-200.

So I implemented the same thing with soccer result. I have like 12 years of data at my disposal, and using these data, I want to predict the soccer result--specifically to its score (ie.2-5,5-7,3-8) not just who wins and who loses.

I treated this as a classification problem.

So my idea was to have a decay rate when evaluating the match. I mean, older matches are pretty much useless right?

I used Bayesian Theorem to predict the result. To get the prior and like hood probability, I used Gaussian and Poisson distribution.

The reason I used Poisson distribution because a research against Bundesliga showed that using Poisson distribution yields best match prediction.

The reason I used Gaussian distribution because it turns out that goal distribution is pretty much log-Gaussian (you have to log transform the data first to make it Gaussian.)

The plan was to train the decay rate using the train data, and later pick the best model using validation data.

I decided to create a variable called decayrate = X, where the value of match occurs in a season is 1/X of the value of the match occurs in the next season.

For ex: I'm predicting a match from 2013/2014 with a decayrate = 1.5. A match from 2012/2013 weights full. A match from 2011/2012 weights = 1/1.5, A march from 2010/2011 weights 1/(1.5^2), and so on.

To find the correct decay rate, I find the logloss of each method when pitted against every decay rate from 1.0 ~ 2.0, multiple of 0.05. The decay rate with the lowest logloss is the optimal decay rate.

As an experiment, I randomly pick 25% match from the pool of data as a train data 10 times--not as a retarded attempt to do cross validation--but to check if the optimal decay rate changes when pitted against different train data.

With Gaussian distribution, the optimal decay rate is always either 1.20 or 1.25--that's close enough.

The problem is when I used Poisson distribution, the optimal decay always changes as I throw in another random training. It can be 1, it can be 2, or it can 1.3. There's no pattern.

So my question is.

1) Is there anything I can do when the data is not working with me?

2) Is there a technical jargon for this? I googled time series but no result. I googled decay rate data science, but what I got was exponential decay rate used for physics.


texts: For stateful LSTM, does sequence length matter?

texts:
With stateful LSTM the entire state is retained between both the sequences in the batch that is submitted, and even between separate batches until model.reset_states() is called. So my question is, does sequence length matter?

E.g. if my entire data set is 1000 rows, and each row has 5 attributes, and I set sequence length of 100 then my batch_input_shape is [10,100,5], and the shape of my Y labels would be [10,3] (3  Y labels, for example)

But as the state is retained between sequences, what difference does it make what length the sequence is as eventually the entire sequence is fed in and the state is retained?

Am I right in thinking that because I specify Y labels for each sequence (10 in this example), then that's the difference?  i.e. it basically defines how many set of labels I can provide for an entire sequence.

For example, if I were to change sequence length to 200 then the batch shape would be [5,200,5] and I'd therefore only provide 5 sets of Y labels for entire batch of sequences. That will change the network's learning behaviour quite substantially, correct?

So, essentially, do I treat it as a hyperparameter and experiment with what sequence length provides the best result?

It's funny with NNs; it seems you need a genetic algorithm to find the combination of hyperparameters that works best.


texts: When unsupervised learning is more beneficial in comparison with supervised learning even the labelings are existed?

texts:
When unsupervised learning is more beneficial in comparison with supervised learning even the labeling are existed?
If there is no labeling the unsupervised learning is better than supervised learning but in some cases even the labeling targets are available, the supervised learning approach works betters? What about conditions of these cases? Can we say that if there is no clear dependency between variables unsupervised learning works better?


texts: Metric to combine average with number of accurance

texts:
I was analysing a dataset which has got two main columns which I can name like:

category:String
amount: Int

So I wanted to know the average of amount for each of the category. So I did something like:

df.groupBy("category").agg(avg("amount") as average)

The problem is for categories I don't have many observations, sometimes just one or two. But for other categories I have around 5000.

So I would like to know if there is any metrics where I can reflect that the average may be more or less accurate depending on the amount of observations used for the average.


texts: What kinds of learning problems are suitable for Support Vector Machines?

texts:
What are the hallmarks or properties that indicate that a certain learning problem can be tackled using support vector machines?

In other words, what is it that, when you see a learning problem, makes you go "oh I should definitely use SVMs for this'' rather than Neural networks or Decision trees or anything else?


texts: "Learning" algorithm to use when future depends on past events (MDP property not met)

texts:
There are around 5 different retirement plans available in my country. People can pick from them freely. I would like to create a solution that would try to predict the best plan(s) given a particular set of initial conditions (representing a person current situation).
My plan was to use Q-learning for this, however the problem I had encountered is that future rewards seem dependant on past actions:

Some benefits (for example reduced tax rate during "cash-out" at the beginning of retirement) will be available only if the agent had chosen to invest in particular plan at least X times in the past.
Investment return will depend on how much was invested in particular plan in the past.

I believe that I could solve the second problem by calculating a reward based only on last decision, but I see no way around first case.
My question is: Am I correct that q-learning, or any other reinforcement learning algorithm, won't be able to take into account those conditions (especially the first one)? If so, would you have any suggestion for AI algorithm that could find the optimal solution in such scenario?
I highly appreciate your help!


texts: Best practices for selecting categorical features

texts:
I'm trying to create a classifier that will predict whether someone will attend an interview or not. Each data point is for a single candidate and contains details such as the location of the interview, the candidate's current location, the job skill requirements, interview type etc. All of the data is categorical.

There are also some features, which you may not think would have an impact on the candidate's attendance e.g. marital status. My initial thought was, based on "common sense" (I use this term loosely), to drop this feature, but I wanted to take a more rigorous approach to determine the importance of this. However, I am unsure of what is best practice. How do I go about determining whether this is a feature that can be removed or not?

Secondly, is there a downside to keeping it even if it has limited predictive power? Aside from creating a more cumbersome model, could it adversely impact the accuracy of a prediction?

Thanks.


texts: Coordinate System's influence on $L$ distances (Manhattan and Euclidean)

texts:
I don't understand this picture, which says if we change the coordinate system, we would have the same result for $L_2$ distance, whereas, our result would differ for $L_1$ distance.
What does it mean by coordinate system? $(0,0)$ if yes, the assertion is not true.

I mean, suppose we have a picture with this matrix A, and another with B, for calculating their L1(Manhattan) and L2(Euclidean) distances, we would have the following code, how is this slide applied to the proposed problem?

import numpy as np
A = [[0,21,2],[3,4,5],[6,7,8]]
B = [[5,6,37],[8,0,10],[11,12,13]]
L1 = np.zeros((3,3))
L2 = np.zeros((3,3))
C = np.zeros((3,3))
for i in range(len(A)):
    for j in range(len(A)):
        L1[i][j] = np.abs(A[i][j] - B[i][j])
        L2[i][j] = np.power((A[i][j] - B[i][j]),2)
sum(sum(L1)),np.sqrt(sum(sum(L2)))


texts:
I have started learning ML and I am stuck at finding a problem solution. I need the steps to follow up:
Below are the formats of data sets:
    cust_Demographics

Customer_ID         0
Nationality         0
Income_Range        0
Job_Type            0
Marital_Status      0
Gender              0
State               0
Language            0
Loyalty_Status      0
Age                 0
Points              0
recent_tran_date    0
Recency             0
custSince           0
Frequency           0
Monetary            0
90999

Customer_Transaction

Year                0
Week                0
Store_Code          0
City_Name           0
Transaction_Type    0
Customer_ID         0
Invoices            0
Item_Count          0
Revenue             0
Discount            0
Units_Sold          0
500000

Customer_Store

Region              0
Region_Code         0
Store_Code          0
Store_Name          0
Mall_Name           0
Sales_Per_Day       0
Store_Size_Sq_Ft    0
Customer_Count      0
Total_Revenue       0
38

Test
Customer_ID    0
Store_Code     0
400000

Problem: Need to predict whether a customer goes to a store(new stores) or not.(0/1).

So , I merge Demographics,Transaction and store table.

Q1 : I have total rows= 500000 and added one target column as 1(represents they went for shopping) so all the values are 1's for this column so how should I train to predict the target value.

Q2 : If I merge Test,demographics,Store it is ok but how can I use the features of transaction table. customer->store (A Customer shops for that store)

Or can it be related to the recommendation based problem?


texts: How to utilize user survey answers and the actual usage in forecasting power usage using LSTM?

texts:
I have the pre-trial survey and post-trial survey conducted of around 5000 users for Smart Meter installation.

With this I have power usage reading recorded every 30 min for around one and a half years.

survey csv is like this

User Question 1 Question 2 .......
1000    a           a      .......
1001    b           a      .......
.       .           .      .......
.       .           .      .......
.       .           .      .......
5000    b           a      .......

power usage csv is like this

User date usage
1000 20001 0.003
1000 20002 0.025
.... ..... .....
.... ..... .....
.... ..... .....
.... ..... .....
.... ..... .....
1000 65047 0.52
1000 65048 0.14

I want to forecast power usage of the user based on the past power usage and the survey using LSTM.
How to start with this?


texts: Non-prediction Applications of Machine learning

texts:
Prediction seems to be the dominant theme of machine learning. Most algorithms have fit and predict functions so that model can be created which can predict outcomes or other parameters of interest from new set of features.

What are non-prediction applications of machine learning?


texts: Correctly interpreting Cosine Angular Distance Similarity & Euclidean Distance Similarity

texts:
As an example, let's say I have a very simple data set. I am given a csv with three columns, user_id, book_id, rating. The rating can be any number 0-5, where 0 means the user has NOT rated the book.

Let's say I randomly pick three users, and I get these feature/rating vectors.

Martin: $<3,3,5,1,2,3,2,2,5>$

Jacob: $<3,3,5,0,0,0,0,0,0>$

Grant: $<1,1,1,2,2,2,2,2,2>$

The similarity calculations:

+--------------+---------+---------+----------+
|              | M & J   | M & G   | J & G    |
+--------------+---------+---------+----------+
| Euclidean    |   6.85  |    5.91 |    6.92  |
+--------------+---------+---------+----------+
| Cosine       |    .69  |    .83  |      .32 |
+--------------+---------+---------+----------+

Now, my expectation of similarity is that Martin and Jacob would be the most similar. I would expect this because they have EXACTLY the same ratings for the books that both of them have rated. But we end up finding that Martin and Grant are the most similar.

I understand mathematically how we get to this conclusion, but I don't understand how I can rely on Cosine Angular distance or Euclidean distance as a means of calculating similarity, if this type of thing occurs. For what interpretation are Martin and Grant more similar than Martin and Jacob?

One thought I had was to just calculate Euclidean distance, but ignore all books for which one user hasn't rated the book.

I then end up with this

+--------------+---------+---------+----------+
|              | M & J   | M & G   | J & G    |
+--------------+---------+---------+----------+
| Euclidean    |   0     |    5.91 |    6.92  |
+--------------+---------+---------+----------+
| Cosine       |    .69  |    .83  |      .32 |
+--------------+---------+---------+----------+

Of course now I have a Euclidean distance of 0, which fits what I would expect of the recommender system. I see many tutorials and lectures use Cosine Angular distance to ignore the unrated books, rather than use Euclidean and ignore them, so I believe this must not work in general.

EDIT:

Just to experiment a little, I adjusted Jacob's feature vector to be much more similar:

Jacob: $<3,3,5,1,2,3,2,0,0>$

When I calculate Cosine Angular distance with Martin, I still only get .82! Still less similar than Martin and Grant, yet by inspection I would expect these two to be very similar.

Could somebody help explain where my thinking is wrong, and possibly suggest another similarity measure?


texts: What is significance of Colour-digit MNIST game in paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning?

texts:
My question is regarding the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Can anyone explain what is the significance of Colour-digit MNIST game in the paper? I understand from here that the agent has to choose an action that represents
if (case 1) color of agent 1 == parity of agent 2
or (case 2) color of agent 2 == parity of agent 1.
The reward for case 1 is twice that of case 2. Hence, it optimal case, both agents should learn (case 1) to communicate their parity to other agent via the message and learn to check the received message against its own color. But if this seems to mean that case 2 is just acting like a noise or additional challenge in attaining optimal solution. Even if the case 2 terms did not exist, still the agents had a learn non-trivial communication protocol. Having said that, does case 2 term has any fundamental implications/need?


texts: Transformer: where is the output of the last FF sub-layer of the encoder used?

texts:
In the "Attention Is All You Need" paper, the decoder consists of two attention sub-layers in each layer followed by a FF sub-layer.
The first is a masked self attention which gets as an input the output of the decoder in the previous step (and the first input is a special start token).
The second, 'encoder-decoder', attention sub-layer gets as an input queries from the lower self-attention sub-layer and keys & values from the encoder.
I do not see the use of the output of the FF sub-layer in the encoder; can someone explain where is it used?
Thanks


texts: How do I apply radon and iradon function in a 3D image

texts:
I am using matlab image processing
, I have a 3DBodyphantom image of a CT scan which displays the axial,coronal and saggital slices, I am to apply to the image the radon function to get a sinogram and then the iradon function, I keep getting an error message while trying to apply the radon function within the loop.
This is the code for viewing the slices, how to apply the radon and iradon is what I am trying to figure out, is there a way to apply it or how am I suppose to apply it? My theta for applying the radon is that projections are taken every 2 degrees of rotation.
This is the code I am using;
dim = [256 256 131]; %[256 256 116];
fid = fopen('3DBodyPhantom.bin'); %fopen('output42_atn_1.bin');
x= fread(fid, 'float32');
y=reshape(x, dim);

%to view sagittal slices
for i=1:dim(1)
    imagesc(squeeze(y(i,:,:)));
    pause(0.05);
end

%to view coronal slices
for i=1:dim(2)
  img1=  imagesc(squeeze(y(:,i,:)));
    pause(0.05);
end

%to view axial slices
for i=1:dim(3)
    imagesc(squeeze(y(:,:,i)));
    pause(0.05);
end

fclose(fid);


texts: How to build a simulator for a physical machine given a set of datapoints of its behaviour?

texts:
I have a database with millions of datapoints describing the behaviour of a heat pump. For every second, I know various temperature, pressure, mass flow and power measurements as a response to the signals sent by a controller. In other words, I have records of what the machine is being told to do and what actually does.
I would like to build a simulator that, given a set of artificial inputs (e.g. coming from a web page) attempts to simulate the behaviour of the machinery.
I have no expertise in machine learning, so I am wondering what are the best/most common approaches to takle this kind of situation?


texts: Calculating Univariate and MultiVariate Logistic Regression with Python

texts:
I have a simple data set of a number of variables and a single binary dependent variable.  The data is stored in a data frame.  When I use python's statsmodels.api and logit.fit() on the dataframe I am presented with a table detailing p values and confidence intervals etc for each of the variables.  I need to calculate both univariate and multivariate p values and confidence intervals for each variable, however I am unsure what logit.fit is calculating - multivariate?  If so how do I calculate univariate values - maybe just analyse a single variable at a time?  Sample output below:

==============================================================================
Dep. Variable:           Vehicle        No. Observations:                11540
Model:                          Logit   Df Residuals:                    11515
Method:                           MLE   Df Model:                           24
Date:                Thu, 29 Aug 2019   Pseudo R-squ.:                 0.05443
Time:                        11:57:39   Log-Likelihood:                -7463.8
converged:                       True   LL-Null:                       -7893.4
                                        LLR p-value:                6.082e-166
=========================================================================================
                            coef    std err          z      P>|z|      [0.025      0.975]
-----------------------------------------------------------------------------------------
Red                    0.0084      0.001      6.880      0.000       0.006       0.011
Green                  0.1345      0.041      3.293      0.001       0.054       0.215


texts: Temporal difference learning with a neural network

texts:
Suppose I would like to train a value network $v$ via TD(0).

So my TD target for a time step $t$ equals:

$$R_{t+1} + \gamma v(s_{t+1})$$

If I understand correctly, I just need to use mean squared error, so that $v(s_t)$ becomes closer to this target. However, my network outputs values between $(-1; 1)$ and rewards are from this interval also, so the TD target lies between $(-2; 2)$. Should I scale it before apply learning? What are the consequences of not doing this i.e. training a neural network with target values from a broader interval that it's output? Can we say anything about it from theoretical point of view?


texts: Why do we choose principal components based on maximum variance explained?

texts:
I've seen many people choose # of principal components for PCA based on maximum variance explained. So my question is do we always have to choose principal components based on maximum variance explained? Is it applicable for all scenarios i.e text count vectors(BoW, tfidf..) where number of dimensions are really high.

Does maximum variance means most information about my data in higher dimension is captured into lower dimension?

Usually I'd plot something like this to see the variance explained.

plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Principal Components')
plt.ylabel('Variance ratio')
plt.show()


texts: How to estimate missing values when calculating NDCG

texts:
I would like to compare recommendations methods using NDCG metric on MovieLens dataset.
In ranking problem, the goal is to rank items based on their relevance for user. Ranking models can be learned based on ratings matrix, where each user rates small subset of all items. Ratings for other items are unknown.
Collaborative Filtering methods can be used to create model which generalize training datasets and predict ratings for unrated items.
Let's consider following example on dataset consisted of 5 movies.
User A rated only 3 movies:

movie 1 - 5 stars
movie 3 - 3 stars
movie 4 - 2 stars

Model predict following results

movie 1 - 5 stars
movie 2 - 4 stars
movie 3 - 3 stars
movie 4 - 2 stars
movie 5 - 1 stars

How NDCG@3 should be calculated in this example ? Movie 2 get second best score but it has not been rated by user although it's highly relevant for user A based on other user ratings. Giving movie 2 1 star rating as ground true penalized model because it predicted highly relevant movie which was not rated by user.
Many papers measure model performance on MovieLens using NDCG, but I have not found details how NDCG is calculated. What is the best practice for solving this problem?
Is it good idea to estimate unknown rating value based on movie ratings median or average ?


texts: Naive Bayes implementation: why Laplace smoothing is different from theory?

texts:
Let's have a Naive Bayes Bernoulli classifier with $n_C$ classes and $n_F$ features.

According to the formula in here and here and almost every theory book I could see, Laplacian smoothing means that we take

$$P(x=X|c=C)=\frac{\#\{x=X,c=C\} +\alpha}{\#\{c=C\} +n_F\alpha }$$

but in the implementations of sklearn and this book the formula is actually

$$P(x=X|c=C)=\frac{\#\{x=X,c=C\} +\alpha}{\#\{c=C\} +2\alpha}$$

why is the number of features taken as two? Is there a theoretical reason?


texts: What if outliers still exist after variable transformation?

texts:
I have a variable with a skewed distribution.

I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still exist.

My question is:
Although after transformation, the variable distribution is nearly Gaussian, if there are still outliers, should we still select this transformation?
Or should we decide to use other techniques such as discretization in order to capture all outliers?


texts: I create a corr() df out of an original df. The corr() df came out 70 X 70 and it is impossible to visualize the heatmap... sns.heatmap(df). If I try to display the corr = df.corr(), the table doesn't fit the screen and I can see all the correlations. Is it a way to either print the entire df regardless of its size or to control the size of the heatmap?


texts: How do you ascertain the quality of your data?

texts: How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such reason is we need to meet deadlines for reports & publications. I am wondering if anyone has adopted or come across a method/guidelines that help to find issues within the data (time-saving tips), so we can analyze the data efficiently. Please share your experience, tips, etc.


texts: How to do multivariable regression in Orange3?

texts: Orange3 contains a number of regression widgets, but they all seem to be univariable i.e. one independent variable that correlates to one dependent variable. When I have more independent variables that might influence a dependent variable, how to handle this in Orange3?


texts: BatchNorm vs InstNorm from the perspective of feature distributions

texts:
What I understand so far...

The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift".

Covariance shift is a difference in distribution of features between the training dataset and inference dataset. But this distribution of features actually is a metadistribution: if you view a single data point as a distribution of features, then the covariance shift is describing the difference in distribution of distribution of features. The fact that instance normalization worked makes me think that this view is more accurate, because what instance normalizatoin does is normalize the distribution of features within a single instance, not amongst all other instances within a batch (which is what BatchNorm does).

Question:

The potentially pedantic description of describing distribution of features as meta distribution was leading up to this question: it makes intuitive sense that normalizing features of all instances within a single batch can help covariance shift, since it is directly normalizing the meta-distribution. Then, what is it the implication of normalizing every individual distribution within the meta-distribution (i.e. InstNorm) in terms of covariance shift? I simply don't know enough about stats to predict the behavior of distribution of distribution, when each distribution within the meta-distribution is normalized.

I understand that my question isn't really well formed and explained, so I will fix it if you point out the obscurities.


texts: Detecting original vs. edited (reposted / recompressed) image

texts:
I'm trying to create something to help solve this problem:

My goal is that, given two images where one is an edit of the other, produce a system that outputs which one is most likely to be the original.
I have attempted to solve the problem in the following way.

Collect a data set of random images that one may find in their social media blog. (10,000 images)

Apply random lossy transformations to the image: resizing (with various scaling algorithms), cropping, lossy compression (JPEG)

Preprocess input into training data:

Metadata:

Collect file size, resolution, file format, lossy compression quality

Add jitter (to prevent overfitting by recognizing specific images), jitter varies per 8x8 sample

Output is a fixed number of floats per datum

Input pixels:

Break the image into 8x8 chunks (the size is chosen to coincide with JPEG MCU size)

Select a fixed number of chunks with the most entropy (changes in pixels)

Select a color channel

Divide by 255.0

Output is 64 floats (one per pixel)

DCT values:

Using the pixel data from above, perform a DCT transform

Take the absolute value of the DCT value

Output is 64 floats

Feed the above data into a neural network.
I don't know if it makes sense or not, but I started with a straight-forward model and did a simple hill-climbing hypersearch which added/removed/edited layers, and it produced the following model:

Each sample (and associated metadata) is fed as a single input.

Fit the model to the labels 0.0 or 1.0, with 0.0 representing original (less noise) and 1.0 representing edited (more noise).
Even though these labels don't describe any quality of the image itself, the goal is to use them to indicate some kind of ordering. The idea here is to "pull" the weights towards 0 for images known to have less noise and towards 1 for those with more noise, which I hope will cause the model to output higher values for images which have been edited.

Currently the model's accuracy is 88% (i.e. it correctly produces higher values for our edited versions for 88% of input image pairs).
My question is if there is a better way to represent ordering in this case other than labeling pairs 0 and 1. One potential issue with this is that the model is penalized if it outputs a score >1 or <0 (though currently avoided using sigmoid activation if that makes sense). Also, if there are other obvious improvements in the whole process. Thank you!


texts: analyze the effect of some new changes to business rules on customers retention and sales

texts:
I am trying to analyze the effect of a particular business rule on customer behavior.
Background: I have two call centers operating in my company. One is an in-house call center and the other one is a third party. The incoming calls are handled by these two call centers based on some rules. 2 months before we changed some operational rules after which all the calls will be routed to call center A and then if not attended to call center B. Call center A has been performing well initially, in the sense customers coming through call center A drive more sales and is retained more than call center B. Now, call center A has been getting more calls and the hypothesis is that customers have to wait in the queue for a long time which eventually affects our sales and retention. I am trying to analyze if there is any difference in the sales, and retention since the rule implemented. So what I am looking to do is Pre-dateXXX and Post-dateXXX sales and retention. I tried some visualization which shows some differences.
But I am not sure how to approach it statistically. There are two approaches I have been thinking of:

Build a logistic regression model and see the effect of this variable on the target variable (retention)
Take a sample of data from pre and post and do a Chi-square test on both to understand if there is any difference between call center A and B.

I am not sure if my approaches will actually solve the business problem: does the customer has to wait in queue for a longer time in service and how is affecting retention and sales data analysis?
How do I approach this? Can someone guide me?


texts: What does it mean to say convolution implementation is based on GEMM (matrix multiply) or it is based on 1x1 kernels?

texts:
I have been trying to understand (but miserably failing) how convolutions on images (with height, width, channels) are implemented in software.
I've heard people say their convolution implementation is done using GEMM, or done using "Direct convolution" or done using 1x1 kernels.
I find it very confusing and can't wrap my head around so many different ways it's described everywhere - I thought I understood a typical convolution like pytorch conv2d as a mathematical operation on an image, but what do they mean when someone says they do conv2d using one of the following ways?

1x1 kernels or 1x1 convolution (what does kernel even mean here)
GEMM
"direct convolution"

For doing Convolution using GEMM, what I understand based on this paper is that each of the input-image and filters are converted to 2d matrices using im2col and im2row ops and then these two are simply matrix-multiplied.
The 3d input image (height, width, input-channels) is converted to a 2d matrix, the 4-d kernel (output-channels, input-channels, kernel-height, kernel-width) is converted to a 2d matrix. Or does "GEMM-based implementation of convolution" mean something else? If that's what it means then how is it different than doing "convolution using 1x1 kernels"?


texts: semantic relation or semantic relatedness between terms or phrases

texts:
I am looking for a method that can realize if there is any semantic relation between two terms using ontologies. For example given two terms "kitchen" and "Chef" it can return s.th like chef works in a kitchen and a correlation value (for example 0.7). What about proper nouns or noun phrases? The relatedness of two phrases such as "the biggest defense against the US Navy's Laser Weapon System" and "China's crippling pollution problem".
I would be appreciated if you could give me some references.


texts:
How can I get information about an entity from DBpedia using Python?

Eg: I need to get all DBpedia information about USA.  So I need to write the query from python (SPARQL) and need to get all attributes on USA as result.

I tried :

PREFIX db: <http://dbpedia.org/resource/>
SELECT ?p ?o
WHERE { db:United_States ?p ?o }

But here all DBpedia information is not displaying.

How can I do this and which all are the possible plugins/api available for python to connect with DBpedia ?

Also what will be the SPARQL query for generating the above problem result?


texts:
Let's say I trained a classifier to assign a shop department to a product, e.g.: ALGIDA Cow milk->Diary. It did it on a domain of official product names.
When I applied pre-trained classifier to names of products in a shopping list, I found it doesn't perform that well as in the source domain because people tend to make spelling errors, abbreviate product names, do not include brand names, etc. in their shopping lists.
Simply speaking, the classifier does not generalize well to another domain.
I tried a few techniques where each alone improved classifier accuracy in a target domain:

Pre-processed product names in a shopping list by correcting spelling errors and expanding abbreviations
Pre-processed product names in a source domain before training by removing brand names.
Stemmed product names before constructing TF/IDF vectors
Embedded product names using pretrained USE (Universal Sentence Encoder) model.

Each of those techniques makes phrases in one domain more similar to phrases in another domain, or features build on top of those phrases more similar across domains.
Would you call any of those techniques domain adaptation?


texts: How to fit a model to the V-shaped data?


texts: Tableau - filter using parameters and calculated fields

texts:
I have boolean columns in my dataset that represents labels. 1 for True 0 For false.
Each of those labels is part of a group.
(ex: Group "type" contains "bug"= True/False and "feature"=True/False).
I want to filter data dynamically based on the parameter input
I set my parameter up for all labels accordingly and added it to the filter tab.

Now I want to dynamically filter using this parameter. ie, if user selects type="bug" all values in column "Type:Bug"=0 needs to be filtered out from the view.
I understand this can be done via calculations:

this is what i've tried (but does not work):
I have created a calculated field and added it to filters..
Please help me filter using parameters.


texts: TensorFlow restored model returning constant negative prediction

texts:
I've trained a CNN model using Google Colab with TensorFlow. It is a pretty basic CNN:

def convolutional_neural_network(x):
    weights = {'W_conv1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
               'W_conv2': tf.Variable(tf.random_normal([5, 5, 32, 32])),
               'W_conv3': tf.Variable(tf.random_normal([5, 5, 32, 64])),
               'W_conv4': tf.Variable(tf.random_normal([5, 5, 64, 128])),
               'W_conv5': tf.Variable(tf.random_normal([5, 5, 128, 128])),
               'W_fc': tf.Variable(tf.random_normal([4 * 2 * 128, 1024])),
               'out': tf.Variable(tf.random_normal([1024, n_classes]))}

    biases = {'b_conv1': tf.Variable(tf.random_normal([32])),
              'b_conv2': tf.Variable(tf.random_normal([32])),
              'b_conv3': tf.Variable(tf.random_normal([64])),
              'b_conv4': tf.Variable(tf.random_normal([128])),
              'b_conv5': tf.Variable(tf.random_normal([128])),
              'b_fc': tf.Variable(tf.random_normal([1024])),
              'out': tf.Variable(tf.random_normal([n_classes]))}

    x = tf.reshape(x, shape=[-1, 110, 42, 1])

    conv1 = tf.nn.relu(conv2d(x, weights['W_conv1']) + biases['b_conv1'])
    conv1 = maxpool2d(conv1)

    conv2 = tf.nn.relu(conv2d(conv1, weights['W_conv2']) + biases['b_conv2'])
    conv2 = maxpool2d(conv2)

    conv3 = tf.nn.relu(conv2d(conv2, weights['W_conv3']) + biases['b_conv3'])
    conv3 = maxpool2d(conv3)

    conv4 = tf.nn.relu(conv2d(conv3, weights['W_conv4']) + biases['b_conv4'])
    conv4 = maxpool2d(conv4)

    conv5 = tf.nn.relu(conv2d(conv4, weights['W_conv5']) + biases['b_conv5'])
    conv5 = maxpool2d(conv5)

    fc = tf.reshape(conv5, [-1, 4 * 2 * 128])
    fc = tf.nn.relu(tf.matmul(fc, weights['W_fc']) + biases['b_fc'])
    fc = tf.nn.dropout(fc, keep_rate)

    output = tf.add(tf.matmul(fc, weights['out']), biases['out'], name='pred')

    return output

And when I test the accuracy it returns ~80%.

So I saved the model and restored in my computer to actually use it in practice. Like so:

with tf.Session() as sess:
    new_saver = tf.train.import_meta_graph('cnn_model.meta')
    new_saver.restore(sess, tf.train.latest_checkpoint('./'))
    graph = tf.get_default_graph()
    pred1 = graph.get_tensor_by_name("pred:0")

    print_pred = tf.Print(pred1, [pred1])
    pred = tf.greater(print_pred, 0.5)
    x = graph.get_tensor_by_name("x:0")
    with driver as dv:
        while(True):
            image = np.array(sct.grab(coordinates))
            image = image[::, 75:615]
            image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
            image = cv2.resize(image, (0, 0), fx=0.1, fy=0.2)
            first_image = np.array(image, dtype='float').reshape(1, 4620)
            if(sess.run(pred, {x: first_image})):
                actions.send_keys(Keys.SPACE).perform()

The output of the CNN is turned into a binary "press or not press" the SPACE key.

My problem is, when I run this, the prediction always returns this value: -0.21995841

Which is strange given that I'm feeding very similar images as the test and train data sets.

Can someone shed a light on what I'm doing wrong?

Thanks in advance!


texts: optimal minimization algorithm for platou

texts:
Greeting, I'm trying to solve an optimization problem (minimization, to be specific).

My problem is that my function has one major plateau (see example image).
I'm using the optimization algorithms of scipy, and only basinhoppin actually solves the problem, but VERY slow.

Can you assist in finding a way to speed things up?
thx


texts: How to reflect different "configurations" of a problem for training an ANN

texts:
I have a regression problem with a number of inputs. The problem also has several "configurations" which change the inputs drastically. But within the same configuration the inputs are changing much less.

Older research has dealt with the problem by training a separate neural network for each configuration. I have used the approach to one-hot encode the configuration and feed them into the neural network as inputs.

This works well, but i wonder if there might be a better way of doing this. I have only tried a multi-layer perceptron. Would a CNN, RNN, LSTM be of use to inherently separate the different configurations?


texts: How to train a single Artificial Neuron to predict f(x), where f(x)=x

texts:
Conditions : 1. Network architecture: one single neuron.
2. x=99 (real number)
3. input to the neuron is x, output must be f(x) after trained.
4. use one bias and one weight.
5. use MSE for calculating error.
6. use gradient descent(original version) for back propagation.
7. learning rate: 0.01, no decay, no momentum.
8. must assign random value of weight and bias
will it work for this problem?


texts:
Recently I am working on the neural network deep learning algorithms, just curious to ask is it possible to merge two neural network models and to output one model that contains all the learned knowledge from the two models?
For example:
Model A can work on feature [A, B]
Model B can work on feature [C, D]
(Model type is the same for model A and model B if this makes things easier.)
After model merging, we get model C which is capable of working on feature [A, B, C, D].
I have looked into the transfer learning and Siamese network, but to my knowledge, I don't think these techniques can help me achieve my goal? (please correct me if I am wrong).
So any ideas will be much appreciated for merging the two models, or any possible technique, terminologies are welcomed.


texts: Scalable Infrence Server for Object Detection

texts:
I have created a Django service (nginx + Gunicorn) for object detection models.
For my case i have 50+ models with resnet 50 based back bone.
Server Machine Specification:

16 CPU
64 GB Ram

I have pre-loaded all the models in my service. I am running 20 inference requests in parallel.
But issue i am facing that Gunicorn intermittently restarts any worker out of 8 workers  when inference is running and it is not due to time out. it starts to reloading the model in service again. Due to this my inference requests are failing.
Can you please Suggest the solution or other way to run inference as service ?


texts: Is there an oriented clustering algorithm?

texts:
I'm looking for a clustering algorithm that will make cluster depending on a orientation. The DBSCAN algorithm cluster points based on a constant radius :
https://upload.wikimedia.org/wikipedia/commons/thumb/a/af/DBSCAN-Illustration.svg/800px-DBSCAN-Illustration.svg.png
Is there a implementation of DBSCAN that is based on "ellipse instead of circle" ?
EDIT: MY SOLUTION
Ok so my solution was to work on my data set.
I had a set of 2D points and I wanted to favor the definition of clusters depending of a given orientation.
My solution was to center my set of point on the origin of the coordinate system, rotate them by the orientation you want and apply this vector field on the set of point : X(x, y) = (x-x*a, y) , where a is the factor that determine if the orientation should matter a lot or not (a ∈ [0, 1]) .
Then apply the DBSCAN of this modified dataset.
I hope I was clear enough, don't hesitate to ask me if it's not the case.


texts: using LSTM encoder decoder framework to generate sentences

texts:
i am trying out multiple approaches to generate domain specific data (sentences).For e.g. if i am building a data set to train a chatbot in the travel domain, i would like to take pairs of sentences like "i would like to book a return trip to Philly commencing on the 24th" and "i am traveling to Phialdelphia for a conference for 3 days beginning the 24th" and use these as the enc and decoder inputs. So once the training is complete, i can feed it the first sentence and generate multiple samples, set a threshold (rather than argmaxing it) and have a fairly large corpus of similar sentences that i could train another classifier with. I personally think its a good analogy to machine translation tasks (where the sampled output is another language but in some sense semantically the same sentence)

While i am trying it out, can anyone tell me if they have tried this approach and seen decent results ? Also any recommendations/ pointers on better approaches? thanks a ton!


texts: Why the my Tensorflow code just use one GPU when I assign more than one

texts: I am trying to run my code on a supercomputer with 8 gpus. Though, I assign 8 gpus but one of them is just occupied. I read some notes in websites and it seems that Tensorflow automatically use gpu if it is applicable but still I don't know how I can use all the gpus. The code is just a deep network to be trained using model.fit() and then predict the test data using model.predict()


texts:
I have daily number of sales prediction. I would like to go to an hourly level, taking this prediction. I have past data at hourly level. At this stage, I do not want to predict at hourly level since my first tests gave large errors. I tried doing a profile matrix that allocates the daily forecast into hourly buckets taking past behaviour, but this doesn't seem to work. I also took the cumulative function, and it improves the results, but still with errors larger than 40%. Do you guys have any other ideas?
Thank you.


texts: New to ML and NLP. Is topic/subject extraction a good place to start?

texts: I'm new to AI and specifically NLP, I always love to study things as part of a random project I decide to build, right now I'm working on a chat bot and I was looking for an easy way(in terms of code) to extract the main topic/subject from a sentence(in English), the problem is that wherever I search, even if it's a simple code, it combines a full sentence "meaning"(device parts of speech), I don't like to implement something I don't understand so a small step for how to extract only the subject will be really good and can help my bot with some directions, Is there a simple code(don't care what language it written with) to extract, only main topic/s from a sentence(in English)?.


texts: Bidirectional Encoder Representations from Transformers in R

texts:
Can anybody suggest to me, where I can find example code for R language for BERT neural network for text mining tasks. All I can see are python examples, and I need R.

https://github.com/google-research/bert
https://github.com/facebookresearch/XNLI
https://github.com/tensorlayer/seq2seq-chatbot
https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/
https://github.com/huggingface/pytorch-pretrained-BERT и чат ботов
https://stanfordnlp.github.io/coqa/
https://github.com/nyu-dl/bert-gen


texts: Convolutional neural network with Deep Q learning in games

texts:
What does

"the average magnitude of maximal action value output by the network"

tell us? I mean if we plot this graph, is it good to start as low value and then increase until it goes in a steady state?


texts: Remove part of the string in R - dropping some words from it

texts:
I have a column in dataframe which should represent the date, named "taken", and looks like this:

taken

Sat Jan 01 15:30:27 CET 2011

Thu Jan 06 00:54:26 CET 2011

Sat Jan 08 19:49:02 CET 2011

Sun Jan 16 09:57:16 CET 2011

etc (more than 30000 rows)

Now, I need to change class from factor to date, but I haven't found solution for this kind of format. Since it would be easier with date looking like this:

taken

Jan 01 2011

Jan 06 2011

Jan 08 2011

Jan 16 2011

...How could I drop some words from a string (obviously, Mon/Tue... time and CET in this case) to change it into date?


texts: Very Deep Convolutional Networks for Text Classification: Clarifying skip connections

texts:
Question RE this research paper if anyone has experience with CNN's, pooling & skip connections: https://arxiv.org/pdf/1606.01781.pdf

In figure 1, the input to the first convolutional block has shape (batch_size, 64, s)
The output from block 1 must be (batch_size, 64, s)
The output from block 2 must be (batch_size, 64, s)

However the output from the pooling step has shape (batch_size, 128, s/2). How can a pooling step increase the number of parameters on axis 1 from 64 to 128?!

My guess is that the input to the pooling layer is actually a concatenation of 2 of the previous layer's outputs. In this case it would have input shape (batch_size, 128, s). However, the paper does not appear to clearly specify at what outputs are concatenated...

Can anyone clarify how this is the case?


texts: How is the validation set processed in PyTorch?

texts:
Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the test set. The ML algorithm is a neural network.
The training set is processed (in minibatches) by the code below. First, one sets the gradients to zero. Then, the model makes a prediction, and the loss is calculated. Next, the gradients are computed, and the weights are updated via backpropagation.
def train(data, label):
    model.zero_grad()
    prediction = model(data)
    loss = loss_function(prediction, label)
    loss.backward()
    optimizer.step()
    return loss

As I understand, the validation set is used for hyperparameter tuning, whereas the test set is used for evaluation of the final model (as a reference to compare performance to other models). The accuracy on the test set is measured after "freezing" the model, like in the code below.
for parameter in model.parameters():
    parameter.requires_grad = False
model.eval()

So, my questions are:

When processing the validation set, is it correct to use the code of
train() or must one omit the backpropagation?

If one assumes that a neural network applies dropout, is dropout
enabled or disabled while processing the validation set?


texts: Is it possible to ensure that all classes are represented in the output of a scikit-learn decision tree?

texts:
I am working with an ordinal classification problem with six ordered classes and I want to compare a neural network classifier with a baseline classifier that is as simple and parameter-free as possible.

In my case, I want the baseline to use only the most important single feature $X$ that the neural network uses. For this particular problem, it makes sense to look for a set of class thresholds $\{t_i\}$ that lets me classify as:

      X <= t_0 -> class 0
t_0 < X <= t_1 -> class 1
t_1 < X <= t_2 -> class 2
t_2 < X <= t_3 -> class 3
t_3 < X <= t_4 -> class 4
t_4 < X        -> class 5

I have looked at a few options, and decision trees seem to fit the bill. They are very simple, and do not require choosing any free parameters (unlike e.g. $k$-nearest neighbours which requires choosing a value for $k$). If I create the tree with the argument max_leaf_nodes=6 it is quite easy to extract the thresholds $\{t_i\}$ from the resulting decision tree after fitting. (If there is another method I could use that would fit the bill as well or better to achieve this, please let me know in the comments!)

I have divided my data into six folds while ensuring that the class distribution is very similar in all folds. For this baseline I balance the classes by oversampling, as the classes are originally somewhat unbalanced. (The ratio between samples in the most common class and the rarest class is around 5:1).

For five out of my six folds, the decision tree method works very well when I use that fold for testing and the rest for training. However, for the last combination of folds used for training, I get a decision tree where class 2 is not represented in the output. Instead, two of the leaf nodes represent class 1:

Is there any way to force the decision tree to build itself in such a way that the six leaf nodes of the tree represent the six different classes?


texts:
I am hoping for a bit of guidance from experienced practitioners / academics.

I want to work through the Bishop ML book, but have minimal background.

What is the fastest way to get the pre-requisites (specific books would be appreciated)?

From searching around I found this potential self-study path:

Statistical Inference - Casella / Berger
Probability Theory and Examples - Durrett
Linear Algebra - Hoffman / Kunze

I checked these books out from the library, but they will take me over a year to work through thoroughly, so it does not seem to be practical.

I have searched around on the internet, but most of the advice doesn't list any specific books, just what subjects I should learn.

About me

Graduated in an unrelated discipline many years ago
Willing to dedicate many hours to this (I am doing this to build a background for a degree in machine learning)
I can code pretty well due to my job


texts: GraphViz not working when imported inside PydotPlus (`GraphViz's executables not found`)

texts:
I've been trying to make these packages work for quite some time now but with no success. Basically the error is:

GraphViz's Executables not found

EDIT: I had not posted a terminal log with the error originally. I'm using Ubuntu now so I won't be able to reproduce the exact same error I got in the past (a year ago, so far away in the past...). However, I've been experiencing a similar --- if not the same --- error in my current setup; even while using a virtual environment with pipenv. The error seems to come from lines that were described in @张乾元's answer:

Traceback (most recent call last):
  File "example.py", line 49, in <module>
    Image(graph.create_png())
  File "/home/philippe/.local/lib/python3.6/site-packages/pydotplus/graphviz.py", line 1797, in <lambda>
    lambda f=frmt, prog=self.prog: self.create(format=f, prog=prog)
  File "/home/philippe/.local/lib/python3.6/site-packages/pydotplus/graphviz.py", line 1960, in create
    'GraphViz\'s executables not found')
pydotplus.graphviz.InvocationException: GraphViz's executables not found

I've tried to install GraphViz via 2 different ways: via pip install graphviz and through the .msi package (and also tried to install pydot, pydotplus and graphviz in many different orders).

The code I'm trying to run is simply a dot-to-png converter for the Iris Dataset.

from sklearn.tree import DecisionTreeClassifier
import sklearn.datasets as datasets
from sklearn.externals.six import StringIO
from sklearn.tree import export_graphviz

import pandas as pd
import pydotplus

from IPython.display import Image

iris = datasets.load_iris()
df = pd.DataFrame(iris.data, columns = iris.feature_names)
y = iris.target

dtree = DecisionTreeClassifier()
dtree.fit(df,y)

dot_data = StringIO()
export_graphviz(
    dtree,
    out_file = dot_data,
    filled = True,
    rounded = True,
    special_characters = True
)
graph_1 = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph_1.create_png())

In Jupyter Notebooks and in Atom, the system seems to be looking for GraphViz inside pydotplus, as it points to ~\Anaconda3\lib\site-packages\pydotplus\graphviz.py. Shouldn't it be the other way around?

Lastly, I just want to point out that I've already tried adding GraphViz's path to the system's PATH using C:\Users\Philippe\Anaconda3\Library\bin\graphviz.


texts: Can I use manifold learning to transform the feature set as a substitute of graph kernel of SVC

texts: I just wonder since the manifold learning under scikit-learn has component of graph-based transformation (e.g. Shortest-path graph search under Isomap) I can then transform the feature data set (i.e. measurements of a chemical physical attributes to decide its toxicity) to something that will measure the graph-based measurements.


texts:
I am trying to implement deep learning for image recognition, but I am still confused with cross-validation. Let's say I will use about 100.000-1.000.000 images in total for a binary classification via CNN (transfer learned and finetuned). Hardware is 1 x RTX 3090. My initial idea was to separate all cases and split into training/test set. Thereafter, I will use StratifiedKFold to perform cross-validation on the training set (e. g. 5-fold), in each fold loading pre-trained weights from ImageNet and training a new top, followed by finetuning the remaining weights of the layers. In the end, for all tested models, I will receive n-fold results for validation accuracy that I can average to have a performance measure. Following my initial concept, I would then take the whole training set and retrain my final model to gain the optimal weights (before applying to the hold-out test set). Here, my questions evolved:

When to stop the training on the full training set, when there is no validation set anymore? Use the mean epochs of the early-stopped models during cross-validation? Otherwise the model would probably heavily overfit and not generalize well to the hold-out test set.
Wouldn't it be more appropriate to again split the training this final run and perform early-stopping on the validation accuracy? Or would this be redundant given the cross-validation runs?
Would it be possible to just take the model (including its weights) of the n-fold cross-validations, that had the best performance based on validation accuracy? This would, however, contradict with the concept that cross-validation is apparently not used for parameter tuning, but only hyperparameter selection.


texts: What happens to the left over unpicked data in Random Forest

texts: I believe in Random forest we pick random samples of training data with replacement. My question is there still is a possibility that we might leave some data out. What happens to that. Does it not affect the random forest training? Do we use that left over later at some point.


texts: Are transposed convolutions computed using the Fast Fourier Transform?

texts: In convolutional neural networks, transposed convolutions are represented by the transposed matrix which represents the convolution. But convolutions are not actually computed by creating this matrix, they are rather computed efficiently by applying the Fast Fourier Transform and multiplying in the Fourier domain. Are transposed convolutions also computed in the Fourier domain? If so, what is the exact operation that is used in the Fourier domain?


texts: shifting the last column in the dataframe to the first place

texts: While this question had relative answers, I couldn't find an efficient and robust way of doing it. I have to shift the last column in my data frame to the first without disturbing any of the other columns. How should I do this ? Thanks in advance.


texts: Recognition of Geometric shapes for ridges

texts:
I want to recognize these geometric shapes which are related to each other. For instance, looking at the image of the roof below, just by knowing the existence of the ridges in RED, I know that the ridge in BLUE should also exist (even if it is not visible in the image). If I have thousands of such labelled images, a ML model should be able to learn this as well. However, I can't figure out how to represent this problem?

Label(s) : C, Z

Label(s) : D

Label(s) : C, Z

Label(s) : E, G

Let's call these ridges as lines, as in the 1st example, we have X and Y lines being detected by simple edge detection but not Z because it's not visible. The same way line D is not detected but lines A, B, C are, from example 2.

What I want is that I formulate a ML model that learns from the X and Y that there should be a Z and subsequently D from A, B, C.

I have a data set of such examples where ridges are labeled (the red and blue is just to distinguish, all ridges are labeled with same color).

There are some important things to keep in mind.

The brightness of the image could vary a lot.
The ridge could have any scale or even orientation (within reasonable limits).
The input image is almost always very noisy.


texts: User-Item based Recommendation system with data containing binary data

texts:
I have a data set which contains about 400,000 unique items present on a platform. The users on this platform can like and save this in their own list. Now, I have about 4000 users with their like data which looks like this
| UserID | ItemID |
|  1     |   3    |
|  1     |   4    |
|  1     |   10   |
|  1     |   13   |
|  2     |   3    |
|  2     |   40   |
|  3     |   1    |
|  3     |   23   |

An item will only be present against the user-item pair only when the user likes an item. I am trying to recommend items to other users/users to other users using User-User/User-Item Collaborative filtering but I couldn't find any leads on how to achieve that using python. Any help on this would really be appreciated!


texts: 'loss: nan' during training of Neural Network in python

texts:
I am training a neural network in python. But the accuracy is very low.

               precision    recall  f1-score   support

           0       0.10      1.00      0.19      1536
           1       0.00      0.00      0.00      1517
           2       0.00      0.00      0.00      1464
           3       0.00      0.00      0.00      1504
           4       0.00      0.00      0.00      1535
           5       0.00      0.00      0.00      1468
           6       0.00      0.00      0.00      1503
           7       0.00      0.00      0.00      1499
           8       0.00      0.00      0.00      1503
           9       0.00      0.00      0.00      1471

    accuracy                           0.10     15000
   macro avg       0.01      0.10      0.02     15000
weighted avg       0.01      0.10      0.02     15000

It seems that during training of the first epoch the loss value returns nan.

    epoch 1: 64/2483 loss: 3.520028:   0%|          | 0/39 [00:00<?, ?it/s]/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: RuntimeWarning: divide by zero encountered in log
      after removing the cwd from sys.path.
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in multiply
      after removing the cwd from sys.path.
    epoch 1: 2483/2483 loss: 2.626681: 100%|██████████| 39/39 [00:00<00:00, 63.65it/s]
        mean epoch 1 loss: train:  nan val: 2.449664
    epoch 2: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 62.68it/s]
        mean epoch 2 loss: train: 2.434646 val: 2.449432
    epoch 3: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 63.45it/s]
        mean epoch 3 loss: train: 2.434639 val: 2.449432
    epoch 4: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 63.47it/s]
        mean epoch 4 loss: train: 2.434639 val: 2.449432
    epoch 5: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 64.81it/s]
        mean epoch 5 loss: train: 2.434639 val: 2.449432
    epoch 6: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 63.85it/s]
        mean epoch 6 loss: train: 2.434639 val: 2.449432
    epoch 7: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 63.89it/s]
        mean epoch 7 loss: train: 2.434639 val: 2.449432
    epoch 8: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 66.05it/s]
        mean epoch 8 loss: train: 2.434639 val: 2.449432
    epoch 9: 2483/2483 loss: 2.626581: 100%|██████████| 39/39 [00:00<00:00, 68.74it/s]
............etc

Im trying to normalize the input dan fill the nan input using:

    #fill the nan
    col_mean = np.nanmean(features, axis = 0)
    inds = np.where(np.isnan(features))
    features[inds] = np.take(col_mean, inds[1])

    #normalize
    features = np.interp(features, (features.min(), features.max()), (0, +1))

but the accuracy still doesn't change.


texts: How to estimate the transition probabilities for a Markov Chain when time intervals are non-equally spaced

texts:
I've been given a dataset with a number of observable states. I am trying to apply a Finite State Markov Chain to model the system, but I found that I can't estimate the transition probabilities if the observed states were sampled using different time intervals. How can I find these probabilities?

I will try to make the question a more clear. I have samples collected in random intervals during a 6-month period. This samples represent the "quality" of a system, which is ranked from 0 to 15 in discrete intervals i.e. (0,1,2...15). I need to model de system using a FSMC to mimic the system's behavior. So far, I have estimated the transition probabilities between states using only the frequencies of those transitions using all the samples. I am not interested in modeling the time, however, I am not sure If I can estimate the transition probabilities in such a simple way or if I have to take the time between samples (which in my case is random) into consideration when estimating those probabilities.


texts:
I would like to learn both Python and R for usage in data science projects.

I am currently unemployed, fresh out of university, scouting around for jobs and thought it would be good if I get some Kaggle projects under my profile.

However, I have very little knowledge in either language. Have used Matlab and C/C++ in the past. But I haven't produced production quality code or developed an application or software in either language. It has been dirty coding for academic usage all along.

I have used a little bit of Python, in a university project, but I dont know the fundamentals like what is a package , etc etc. ie havent read the intricacies of the language using a standard Python Textbook etc..

Have done some amount of coding in C/C++ way back (3-4 years back then switched over to Matlab/Octave).

I would like to get started in Python Numpy Scipy scikit-learn and pandas etc. but just reading up Wikipedia articles or Python textbooks is going to be infeasible for me.

And same goes with R, except that I have zero knowledge of R.

Does anyone have any suggestions?


texts: which is better : F1-score of 'N' in imbalanced data or 'N+3' in balanced data?

texts:
I have a dataset with 4712 records. Label 1 is 1558(33%) and Label 0 is 3154 (67%)

a)  Currently when I run the model and analysis as is (without sampling techniques), I get an F1-score of 71-77. I chose F1-score and AUC score as the metric as my dataset is imbalanced. (Atleast that's what I felt looking at class proportion). My AUC also ranges between 80-83 for tree based models. Screenshot of models with imbalanced data is given below

b) When I under-sample the majority class, I get all the metrics like F1-score/accuracy/AUC above 80 but less than 85

screenshot below

Now I am not sure which one should I consider? I know it's all about trade-offs.

My objective is to avoid/minimize the misclassifications

Based on your experience in ML projects, what would you guys suggest?

Can someone enlighten me with some reasons why to choose one over the other?


texts: How to calculate Cumulative Sum with Groupby in Python?

texts:
I am trying to calculate cumulative sum with groupby using Pandas's DataFrame. However, I don't get expected output.

My Source Code:

import pandas as pd
Employee = [['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H'],
        ['CSE', 'CSE', 'EEE', 'EEE', 'CE', 'CE', 'ME', 'ME'],
        ['Cat-1', 'Cat-2', 'Cat-1', 'Cat-2', 'Cat-1', 'Cat-2', 'Cat-1', 'Cat-2']]

index = pd.MultiIndex.from_arrays(Employee, names=['Name', 'Dept', 'Category'])

Scale = [1, 2, 2, 3, 3, 1, 2, 3]
Salary = [100, 200, 200, 300, 300, 100, 200, 300]

df = pd.DataFrame({'scale': Scale,
               'salary': Salary},
              index=index)

df1 = df.groupby(['Category', 'scale']).cumsum()
print(df1)

Expected Output:

Cat-1    1         100
         2         500
         3         800
Cat-2    1         100
         2         300
         3         900

Obtained Result:

Name Dept Category
A    CSE  Cat-1        100
B    CSE  Cat-2        200
C    EEE  Cat-1        300
D    EEE  Cat-2        500
E    CE   Cat-1        600
F    CE   Cat-2        600
G    ME   Cat-1       1000
H    ME   Cat-2       1200

Groupby doesn't work. However, if I use sum() (i.e. df1 = df.groupby(['Category', 'scale']).sum()) instead of cumsum(), groupby works perfectly.


texts: Which technologies and skills should the new generation of data scientists learn?

texts:
Having looked at various job requirements, I see that a lot of them require you to be an expert in Hadoop, use Spark but I am wondering how useful are these technologies (due to my lack of understanding)?

It seems that whatever I need to do, I always am able to solve using Google Cloud. For example, I can train models on huge datasets (I've only tried GBs so far) using Cloud ML. I can use BQ/Redshift for data prep and so on.

So my question is, how future proof are Spark/Hadoop, why are they an important skill to learn in 2018 and will they be replaced/getting replaced already?


texts:
I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get the best results.

My methodology is this:

Split my data into training/test.
Use GridSearch with 5Fold Cross validation to train and test my estimators(Random Forest, Gradient Boost, SVC amongst others) to get the best estimators with the optimal combination of hyper parameters.
I then calculate metrics on each of my estimators such as Precision, Recall, FMeasure and Matthews Correlation Coefficient, using my test set to predict the classifications and compare them to actual class labels.

It is at this stage that I see strange behaviour and I'm unsure how to proceed. Do I take the .best_estimator_ from the GridSearch and use this as the 'optimal' output from the grid search, and perform prediction using this estimator? If I do this I find that the stage 3 metrics are usually much lower than if I simply train on all training data and test on the test set. Or, do I simply take the output GridSearchCV object as the new estimator? If I do this I get better scores for my stage 3 metrics, but it seems odd using a GridSearchCV object instead of the intended classifier (E.g. a random Forest) ...

EDIT:
So my question is what is the difference between the returned GridSearchCV object and the .best_estimator_ attribute? Which one of these should I use for calculating further metrics? Can I use this output like a regular classifier (e.g. using predict), or else how should I use it?


texts: Bag of words: Prediction on new (out-of-sample) data

texts:
I'm working with a bag of words in R:
library(tm)
corpus = VCorpus(textsource)
dtm = DocumentTermMatrix(corpus)
dtm = as.matrix(dtm)

I use the matrix dtm to train a lasso model.
Now I want to predict new (unseen) text. The problem is, that I need to generate a new dtm (for prediction) with the same matrix columns as the original dtm used for model training.
Essentially, I need to populate the original dtm (as used for training) with new text.
Example: "original text" would yield a dtm used for taining:
original | text
1          1

While new (unseen) text, e.g. "new text" should yield a dtm for prediction:
original | text
0          1

Q: What is the most efficient way to populate an existing document term matrix / bag of words with new (text) data in R?


texts: How multi-scale CNN selects final output map

texts:
I read a few days ago about multi-scale CNN (OverFeat method) which you can access to presentation via this link. They performed CNN on different scales of an image and then combine all output maps. They said inside of that presentation:

Classification    performed   at  6   scales  at  test time,  but only    1 scale at  run time    .

So my question is: If we use 6 different scales of CNN architecture, then we have different convolution layers in every scale (I guess so). So how in OverFeat, they use 1 scale in run time? If we use a specific scale, then how can we access other feature extractors of different scales? And I see in the article, they combine feature maps of different scales but I can't figure out how this process performed.


texts: Improve the speed of t-sne implementation in python for huge data

texts:
I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec).
I am using TSNE implementation from sklearn.manifold module for it and the major problem is time complexity. Even with method = barnes_hut, the speed of computation is still low. Some time even it runs out of Memory.

I am running it on a 48 core processor with 130G RAM. Is there a method to run it parallely or make use of the plentiful resource to speed up the process.


texts: Limitations on the number of items to use in apriori algorithm?

texts:
Are there any limitations on the number of items to use in a transaction for applying apriori algorithm.

I have a dataset with just 20 records, but the number of items extend upto 900. I am getting memory issue when I apply the apriori algorithm on the dataset.

Thank you


texts: How should one deal with implicit data in recommendation

texts:
A recommendation system keeps a log of what recommendations have been made to a particular user and whether that user accepts the recommendation. It's like

user_id item_id result
1       4       1
1       7       -1
5       19      1
5       80      1

where 1 means the user accepted the recommendation while -1 means the user did not respond to the recommendation.

Question: If I am going to make recommendations to a bunch of users based on the kind of log described above, and I want to maximize MAP@3 scores, how should I deal with the implicit data (1 or -1)?

My idea is to treat 1 and -1 as ratings, and predict the rating using factorization machines-type algorithms. But this does not seem right, given the asymmetry of the implicit data (-1 does not mean the user does not like the recommendation).

Edit 1
Let us think about it in the context of a matrix factorization approach. If we treat -1 and 1 as ratings, there will be some problem. For example, user 1 likes movie A which scores high in one factor (e.g. having glorious background music) in the latent factor space. The system recommends movie B which also scores high in "glorious background music", but for some reason user 1 is too busy to look into the recommendation, and we have a -1 rating movie B. If we just treat 1 or -1 equally, then the system might be discouraged to recommend movie with glorious BGM to user 1 while user 1 still loves movie with glorious BGM. I think this situation is to be avoided.


texts: Oversampling for regression for data grouped in clusters

texts:
I am dealing with a regression problem in which I want to predict the upcoming value of a time-dependent variable by using the previous values of other variables (not including the output variable itself), and the relations between variables are expected to be complex, so I am using an LSTM Neural Network. The input data come from several different sources, and for each source, the data take values in some specific range. Let me give you a simplified example of what I mean:

Let us have a variable $X$ that we want to use to predict the value of $y$. Each $(X,y)$-tuple can come from 4 different data sources.

From the source $S_1$ we obtain data for which $X$ takes values mostly in the interval $[30,32]$ and $y$ takes values in the interval $[10,12]$.
From the source $S_2$ we obtain data for which $X$ takes values mostly in the interval $[40,42]$ and $y$ takes values in the interval $[14,16]$.
From the source $S_3$ we obtain data for which $X$ takes values mostly in the interval $[50,52]$ and $y$ takes values in the interval $[7,9]$.
From the source $S_4$ we obtain data for which $X$ takes values mostly in the interval $[60,62]$ and $y$ takes values in the interval $[11,13]$.

I am not using the source $S_i$ as a training variable.

If I train a model over this data, it works fine, because it does the following: it learns that if $X$ is in $[30,32]$, then it should return a value in between $[10,12]$, if $X$ is in $[40,42]$ it should return a value in between $[14,16]$, etc. That is, even if I am not saying to the model from which source the data come, it learns the source from the data and acts as if we had a different model for each source.

Of course, if I remove one of the sources, train the model over the data coming from the other three and test it on the one I left out, the predictions are not good, because the model never learned how to predict values in that range.

I know (or I strongly believe) that there are some relations in the variables that would allow building a decent model regardless of the source, but to use such a complex model as LSTM NN leads to overfitting for each source.

Of course, I know that I could try with a simpler, more interpretable model, where the predictions for the input data that fall out of the clusters where nicely interpolated from the ones inside the clusters. But, assuming I want to keep the LSTM model, what options do I have? I was thinking about some kind of oversampling technique aimed at generating training data for the values of $X$ that fall outside the intervals. Is there a good way of tackling my problem?


texts: Can anyone explain me the difference between Factor Anaysis and PCA?

texts:
Is PCA and Factor Analysis same? Both are used for Data dimension reduction but theoretically I am not able to find the difference between them?

I did FA in SPSS to reduce number of variables in my data set. I think the same think can be done using PCA as well. Never did PCA so wanted to know exact difference between two techniques.


texts: How to find 4th point equidistant from 3 other points

texts: For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The values are 1 dimensional points.


texts: How to train millions of doc2vec embeddings using GPU?

texts:
I am trying to train a doc2vec based on user browsing history (urls tagged to user_id). I use chainer deep learning framework.

There are more than 20 millions (user_id and urls) of embeddings to initialize which doesn’t fit in a GPU internal memory (maximum available 12 GB). Training on CPU is very slow.

I am giving an attempt using code written in chainer given here

Please advise options to try if any.