texts: Method for predicting price based on Geographical market, Product, and Company

texts:
I have a dataset which tracks the prices of 21 products, charged by 24 companies, in 150 different cities across the globe. However, the data set has missing values--that is, I might have Company X's price for Product A in London, but not New York. I am looking for guidance on how to impute these values given the data that I already have.

There are trends to be exploited in the data. Such as:

Geographical trends: i.e, London and New York are similar markets as measured by the median price of a given product, but markets like Cairo and Johannesburg tend to be much more expensive (though in varying degrees for different products)
Product Trends: Within a given market, for a given company, products are priced hierarchically, although the relationship is nonlinear. For example, if the products are "Good", "Better" and "Best", the price of "Better" might be 2X"Good", and the price of "Best" might be 2.5X "Good", though these relationships are not strictly held.
Company Trends: Certain companies tend to have higher prices than others in most markets, but again, these relationships are not strictly held, and may differ for certain subsets of products or markets where a company has a competitive advantage.

I describe the method that I have tried so far below, though I feel it is overly simplistic. I am looking for suggestions as to how to better capture the complexity of my data and obtain a reasonable accurate imputation for the missing values. Any advice is appreciated.

The method I have tried so far is to estimate coefficients for each category. For example, holding Company and Product constant, what is the average price multiple of Market B over Market A? Market C over Market A? Then, if I know I have a price for (Company X, product "Good") in markets B and C, I'll multiply them by those average multiples in order to obtain a "best guess based on market". Then I repeat that holding constant Company and Market, and Market and Product. At the end, I'm left with three "best guesses". Then the issue is converting my three best guesses into a single guess, because, based on some testing of this method, the true price may lie either between or outside these three best guesses. This is the histogram of errors based on a training sample:


texts: recommend new category paths based on factor item matrix and sales of the items

texts:
Matrix A be a user item matrix. Upon performing  UV decomposition, I have just the V matrix.
The matrix A differs every week and I get a new V matrix every week. The matrix U is not kept track of and I have no information about it.I have access to the sales of the items every 6 months.
I have to predict which items will sell well and what new items can be added to the list that can possibly sell well.


texts: Regression Model Evalation with unbalanced target; alternative to R-squared?

texts:
The Data look something like this:

[0, ....,  0, 2.8, 10, 4.5,]

So there are a lot of zeros to predict. And just a few non zeroes.

Apllying R2 to a potentially predictive model for this kind of Data is not feaseable:
The difference between all zero prediction and the expected values is not that big.

Is there a R2-like metric to aid this situation ?


texts:
Has anyone tried using Bayesian optimisation to get best learning rates, and other hyperparameters for deeplearning.

How to change the parameters between the training. Any examples on callbacks?

Can I see some codes to implement them?


texts:
I'm working on forecasting daily volumes and have used time series model
to check for data stationarity. However, I'm strugging at forecasting data with 90% accuracy. Right now variation is extremely high and I'm just unable to bring it down.

I've used log method to transform my data. Please find the link to folder below which contains ipynb and csv files:

https://drive.google.com/drive/folders/1QUJkTucLPIf2vjo2mRmoBU6be083dYpQ?usp=sharing

Any help will be highly appreciable

Thanks,
Rahul


texts: SGD versus Adam Optimization Clarification

texts:
Reading the Adam paper, I need some clarificaiton.

It states that SGD optimization updates the parameters with the same learning rate (i.e. it does not change throughout training). They state Adam is different as learning rate is variable (adaptive), and can change during training.

Is this the primary difference why Adam performs (for most cases) better than SGD? Also, it states that it is computationally cheaper, how can this be given that it seems more complex than SGD?

I hope my questions are clear!


texts: Why margin-based ranking loss is reversed in these two papers?

texts:
For knwoledge graph completion, it is very common to use  margin-based ranking loss

In the paper:margin-based ranking loss is defined as

$$ \min \sum_{(h,l,t)\in S} \sum_{(h',l,t')\in S'}[\gamma + d(h,l,t) - d(h',l,t')]_+$$

Here $d(\cdot)$ is the predictive model, $(h,l,t)$ means a positive training instance, and $(h',l,t')$ means a negative training instance corresponding to $(h,l,t)$.

However, in the Andrew's paper, it defines

$$ \min \sum_{(h,l,t)\in S} \sum_{(h',l,t')\in S'}[\gamma + d(h',l,t') - d(h,l,t)]_+$$

It seems that they switch the terms $d(h',l,t')$ and $d(h,l,t)$.

My question is that

does it matter to switch $d(h',l,t')$ and $d(h,l,t)$? it's real strange definition. Thanks


texts:
I have been searching for methods to plot in PySpark.

I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are:

through method collect() which brings data into 'local' Python session and plot
through method toPandas() which converts data to 'local' Pandas Dataframe.

The problem is that these both are very time-consuming functions. Is there any method by which we can plot data residing in Spark session directly (not importing it into the local session)?


texts: How to retrieve column names from applying a wrapper method in feature selection?

texts:
This question probably has a simple answer to it, so I will get to the point...
How do I retrieve the names of the columns from applying a wrapper method in feature selection?
Code I have used:
from mlxtend.feature_selection import SequentialFeatureSelector as SFS

X = df[['A','B','C','D']].values
y = df[['F']].values

classifier = KNeighborsClassifier(n_neighbors=7)
code = SFS(classifier,
           k_features=5,
           forward=True,
           floating=False,
           verbose=8,
           scoring='accuracy'
)

code.fit(X, y)
code.k_feature_names_

Output:
Features: 3/3 -- score: 0.78

('1' '2', '3')

Therefore, how do I retrieve the names of the features ('1' '2', '3')?


texts: R - Order each matrix inside a list of matrices

texts:
I have a list where each element is a matrix, and I want to order each matrix individually

Order a matrix is as easy as this:

data <- data[order(data$value),]

But I am struggling to order a list of matrix individually

I have tried this with  some variations:

 for(i in 1:lenght(data)){
 data[i] <- data[i][order(data[i]$value),]
 }

data is the list that contains matrices

I have also tried with apply functions

Some ideas?

Thanks!!

All the code:

#The aim is to show one hospital for each state given an outcome
#in the posotion num of the ranking, its a learning exercise

rankall <- function(outcome, num = "best"){

if (outcome == "heart attack"){
  col <- 11
}else if(outcome == "heart failure"){
  col <- 17
}else if(outcome == "pneumonia"){
  col <- 23
}else{
  stop("invalid outcome")
}

data <- read.csv("outcome-of-care-measures.csv", colClasses = "character")

data <- data[, c(2, 7,col)]

colnames(data) <- c("Hospital", "State", "value")
data$value <- as.numeric(data$value)
data <- data[complete.cases(data),]
data <- split(data, data$State)

#data2 is the final result dataframe
data2 <- data.frame(matrix(nrow = 0, ncol = 3))
colnames(data2) <- c("Hospital", "State", "value")

#From here is what I do not know how to do it.

#data is a list of matrices, each matrix corresponds to a state
#data2 is the result dataframe, where I am going to add the row num of each
matrix in data

#1. Order each matrix in data by value and alfabetically if tie
#2. Add to data 2 the row num of each matrix in data once they are sort

for(i in 1:lenght(data)){

data[i] <- data[i][order(data[i]$Hospital),]
data[i] <- data[i][order(data[i]$value),]

if(num =="best"){
  num = 1;
}else if(num == "worst"){
  num = nrow(data[i])
}

num <- as.numeric(num)

rbind(data2,data[i][num])

}

return(data2)

}

Sorry about identations, it changed when I paste the code


texts:
I am trying to process a large set of location data where a list of start and end coordinate is given. For example,

[
  [(start_lat1, start_lon1), (end_lat1, end_lon1)],
  [(start_lat2, start_lon2), (end_lat2, end_lon2)],
  [(start_lat3, start_lon3), (end_lat3, end_lon3)],
  [(start_lat4, start_lon4), (end_lat4, end_lon4)],
  [(start_lat5, start_lon5), (end_lat5, end_lon5)],
  ...
]

My goal is to create clusters so that if different pair of start and end locations are close, they will form a cluster of pair with those start and end locations. For example, the average for the clustered pair will look something like this,

[
  [(start_lat_C1, start_lon_C1), (end_lat_C1, end_lon_C1)],
  [(start_lat_C2, start_lon_C2), (end_lat_C2, end_lon_C2)
  ...
]

I was following https://geoffboeing.com/2014/08/clustering-to-reduce-spatial-data-set-size/ this, but the tutorial only works for a single coordinate point. Any help about how to approach for paired clustering would be much appreciated.


texts:
I would like to know how I can create a sentiment model from scratch.
I have my data, list of texts, with no labels about sentiment.
Author     Quotes
Dan Brown  “Everything is possible. The impossible just takes longer.”
Dan Brown  “Great minds are always feared by lesser minds.”
Dan Brown   “Men go to far greater lengths to avoid what they fear than to obtain what they desire.”
Dan Brown   “Google' is not a synonym for 'research'.

etc...
I have 20000 of quotes like the above, from other authors too.
My dataset is the only set of data that I have, so I would need to split into training (80%) and test set (20%).
I should work on my training set to build the vocabulary of 'sentiment'.
What I have thought is to clean the text removing stopwords, except negative words (like no, not, ...). Then look for adjectives and assign them a score manually (?).
Once done this, I should have also consider n-grams, especially with negative stopwords, in order to create a small dictionary which could take into account also this.
However I do not know if it could be the right approach, if it is something insane...
Is there any way to build from scratch a sentiment model? How could the 'machine' learn from this?
I would need to develop a model from scratch as I would like to analyse texts in other languages (like Italian or Spanish) and there are no models (not good model at least) in Python to do that.


texts: Is it possible to identify what data a neural net was trained on?

texts:
For example, if there are 5 major data sets for a certain topic. Would it be possible to see if a deployed network was trained off of one them?

How would one approach this?


texts: Which nosql db is better for linking two json files to model for ml.net?


texts: Feature engineering keywords for keras network

texts:
I am building a multi-label classifier, using the keras sequential API. My goal is to take an input of keywords 'cat', 'dog', 'rabbit' and apply a custom label.

My first attempt was to one hot encode all of the keywords, this works fine for training, but when doing online predictions - in another system the data processing becomes tough, as there is no easy way to maintain the positions of keywords on the input to the NN.

My idea is to therefore convert all the keywords to ints and feed this to the network. Is this feasible?

The data structure is as follows:

| label | keyword_array          |
| 123   | ['dog','cat', 'happy'] |
| 234   | ['sad','foot', 'fast'] |

New keywords are also constantly being added as the model is retrained, also an online prediction could have an unseen word.

EDIT:
Ideally I could use a system that given a string of words, or an list - would return some representation which I could feed into my network both for training and prediction.


texts: Training Doc2Vec and Word2Vec at the same time

texts:
As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed.

If my goal is that conceptually similar vectors (regardless of whether they represent documents or words) are close to each other, then would it be better to train them simultaneously?

For instance, a document about Godzilla should be similar to the word Godzilla (as well as to other documents about Godzilla) as well as to the word Japan.

Furthermore, could such a result being achieved with a typical implementation, when including single words as documents into the training data?


texts: Kappa near to 60% in unbalanced (1:10) data set

texts:
As mentioned before, I have a classification problem and unbalanced data set. The majority class contains 88% of all samples.
I have trained a Generalized Boosted Regression model using gbm() from the gbm package in R and get the following output:

  interaction.depth  n.trees  Accuracy  Kappa  Accuracy SD  Kappa SD
  1                  50       0.906     0.523  0.00978      0.0512
  1                  100      0.91      0.561  0.0108       0.0517
  1                  150      0.91      0.572  0.0104       0.0492
  2                  50       0.908     0.569  0.0106       0.0484
  2                  100      0.91      0.582  0.00965      0.0443
  2                  150      0.91      0.584  0.00976      0.0437
  3                  50       0.909     0.578  0.00996      0.0469
  3                  100      0.91      0.583  0.00975      0.0447
  3                  150      0.911     0.586  0.00962      0.0443

Looking at the 90% accuracy I assume that model has labeled all the samples as majority class. That's clear.
And what is not transparent: how Kappa is calculated.

What does this Kappa values (near to 60%) really mean? Is it enough to say that the model is not classifying them just by chance?
What do Accuracy SD and Kappa SD mean?


texts: How to remove outliers from each group within a boxplot?

texts:
I am wanting to remove all the outliers for each individual ID in a boxplot. I have tried the following code:
'''
    AB_Q1 = combine['AB_MEAN'].quantile(0.25)
    AB_Q3 = combine['AB_MEAN'].quantile(0.75)
    AB_IQR = AB_Q3 - AB_Q1

    AB_filter = (combine['AB_MEAN'] >= (AB_Q1 - 1.5 * AB_IQR)) & (combine['AB_MEAN'] <= (AB_Q3 + 1.5 * AB_IQR))
    AB_remove = combine.loc[AB_filter]

    combine.drop(combine.loc[AB_filter].index, inplace=True)

    AB = sns.boxplot(x=combine['FEEDER'], y=combine['AB_MEAN'], hue='FEEDER', data = combine)

'''
This worked for removing all the outliers for the particular column of data but removed too much needed information.
The first images is similar to the original data plot before the data was removed.

The next image shows what happened after the data was removed.

As seen in the first image there are 6 unique identifiers with 4500+ rows of data apiece.


texts: How to model predict on Tensorflow model that has feature columns?

texts:
So far all the coding examples using feature columns do not have examples of how to format their model.predict(...).  I tried using raw string and also putting them into np.arrays.  I am very new so please be gentle :).
Here is the official page, as you can see, no predict example:
https://www.tensorflow.org/tutorials/structured_data/feature_columns


texts: LDA-like algorithm but at the character level?

texts:
I have a catalog of products and I'd like to find "topics" in their description. The problem is that you might find in the description things like GraphicsCardVendor10.2 GraphicsCardVendor_10.3 etc. Obviously, for us humans it is simple to unravel the topic "Graphics Cards"
Of course, I can do some preprocessing but I'm looking for a more holistic way to deal with that. I know some algorithms (like RNNs) can work on character-level.
My end-goal is to find patterns in the data. Sort of, points of interest.
Like, "Hey look, I see a lot of graphics cards over here".
I'd like to get your suggestions what should be the right model for this problem.

Note: The data is currently not labeled.


texts: Using categorical and continuous variables in Deep Learning

texts:
I would like to apply a MLP to some business seller data. I found that the data is a mix of both categorical and continuous features. For what I read it is not advisable to feed a neural network with both types of data (reference unknown/unavailable) and I remember that I read that one can use the following model:
Categorical variables-->NN model 1
                                        ----->NN model 3---->Output
Continuous variables--->NN model 2

So in this model we have two neural networks that are fed each one with only categorical or continuous variables, and then the outputs (from both models) are feed to the third model.
Side note: For what I can see in this proposed model I can end up with an endless "loop", because maybe the output from the model 1 is categorical and the model 2 output is continuous. (?)
My question is how can I model data with a mixture of categorical and continuous features using deep learning (i.e. Neural Networks)? I do not want to use a random forest or any form of decision tree.
Thanks in advance.


texts: Hi I am studying tensorflow for cifar-10 image classification using the code here


texts: How to deal with unknown classes with a neural network classifier?

texts:
I have a small RNN with a softmax output, which succesfully classifies sequences within a known set of n classes. The model is only trained with known classes.

Now I have the problem that there might be sequences which do not belong to any of said classes.

Of course, my model will try to lump in those sequences into one of the given n categories.

How to deal with these sequences that do not fit into any of the given/trained categories?


texts: Pre trained dataset for Car damage detection

texts:
I'm making a Car Damage Detection model which would have 2 classes to detect upon. My dataset has a total of 300 images (out of which I'd be using some for testing), which are totally insufficient to train the model from scratch.

Can I use a pre-trained model to train my dataset and detect images based on the 2 classes, and if yes, then which one should be the best on my problem set?

P.S. - I would prefer to use Faster-RCNN on my dataset


texts: How do I find the eigenvectors corresponding to the largest eigenvalue of a matrix in scikit?

texts: Im trying to determine the principal component 1 and 2 of a symmetric matrix using sklearn. Id appreciate any help. Thank you.


texts: Is backpropogation used in convolutional neural networks?

texts: Do convolutional neural network use the backpropogation algorithm? I am not understanding what exactly happens in fully connected layers?


texts: Data simulation using make_classification in Python

texts: I have a question about data simulation in Python. I deal with the classification of imbalanced data and want to test the effectiveness of different methods on simulated data. I have seen in various articles and books that the make_classification function is used to generate data. Then the data is generated from a normal distribution, so the data is continuous and not discrete. Are such data correct for classification (SVM, Decision Trees) research?


texts: What does it mean when an Actual vs Predicted plot is like this assuming you are using the best model?

texts:
I'm trying to predict the monetary value in a fixed time-frame for a project. I wanted to start with a baseline model before doing any feature engineering or advanced pre-processing.

I'm using a feed-forward neural network for regression (Sklearn's MLPRegressor) and I'm exploring a normal, wide and deep neural network.

Usually what I do is feed a dictionary of parameters to my grid-search and get my baseline model from there. Surprisingly enough, the Actual vs Predicted plot I'm getting this time from my best model is confusing.

This is the plot I'm getting here, red is actual and blue is predicted :

This is the loss curve for that model, spoiler alert, it's like I've never seen before :

And these are the parameters from the best model :

{'activation': 'relu', 'alpha': 0.05, 'learning_rate': 'adaptive', 'learning_rate_init': 0.001, 'hidden_layer_sizes': (24, 12, 6), 'solver': 'adam'}

What am I doing wrong ?


texts: Why study properties with infinitesimal change?

texts:
I read about analysis on local properties of neural networks. Some of them study the impact of "infinitesimal" change to an input. Like in Percy Liang's paper Understanding Black-box Predictions via Influence Functions. They have neat formalization with these properties but I don't really get the point.

My questions are:

How local is local? Or maybe how small is "infinitesimal". I get the idea that this is related to some derivative and stuff. But these results are applied to some changes that are not that small from my perspective, like a change of 0.1 in pixel value ranging from 0 to 1. Why is this considered correct?
From my understanding, if the derivative does not change much, it can be applied to a larger area. But is this the case in neural networks? It would be great if I can get some references on this topic.
Since I don't really have much background in ml theory and serious math stuff. Other pointers that may help me understand these topics are also appreciated.

Thank you for the help in advance.


texts: Searching interactions with RandomForest and/or GBM

texts:
I'm trying to explain a count variable and a continious variable > 0 with GLM, using R.
In order to improve the quality of the regression, I want to add some interactions that can be useful for the model.
As I'm a newbie in machine learning, I want to know if RF and GBM can help me to determine useful interactions.
I saw that interact.gbm can assess the relative strength of interaction effects in non-linear models. The question is :
Will it be "mathematically" correct to add variables with important strength of interaction in order to reduce MSE/Deviance ?

Thank you !


texts: Data Leakage when preprocessing categorical features?

texts:
I am fairly new to machine learning. I came across the concept of Data Leakage. The article says that always split the data before performing preprocessing steps.
My question is, do steps such as discretization, grouping categories to a single category to reduce cardinality, converting categorical variables to binary variables, etc. lead to Data Leakage?
Should I split the data to train and test set before applying these steps?
Also, which are the main preprocessing steps I really need to be cautious of in order to avoid data leakage?


texts: Convolution operator yields negative index of matrix

texts:
When I read about convolutional neural network from the internet, like this one, mostly I found that discrete convolution operator is defined as follow:
$$C=I*F$$
$$C(x,y)={\sum_{a=0}^{k-1} }{\sum_{b=0}^{k-1}}I(x-a,y-b)F(a,b)$$

Where the size of $F$ is $k\times k$, Suppose that the size of F is $3\times 3$ and size of $I$ is $9\times 9$, then $C(1,1)=\dots +F(2,2)I(1-2,1-2)+\dots$ that is does not make sense when $I$ has negative index, how do I compute matrix $C$ ? does we change the way the matrix indexed?


texts: Emotion Recognition with Multi-task Learning

texts:
I am beginner in Data Science and currently working on a learning project aimed at emotion recognition from a bio-medical sensor dataset. I want to use a Multi-task Learning to generate the output of three emotion levels - Arousal, Dominance & Valance.
The dataset consists of 8 sensors data from 20 subjects, here I have attached a screenshot of very small part of the dataset to give youbetter insight about it:

So as you can see, the Dataset is about:

Multiple subjects and for each subject there are 11 columns of data.
Columns 1-8 are raw sensor data (EEG, EOG, GSR, Skin Temperature, IR Response,... ) at 250 Hz.
Columns 9-11 are the corresponding labels.


texts:
I have been working on data science project where I am trying to build a metric for how inbred a source is in a network. We hope to apply this to intelligence reporting, in which documents tend to cite other sources without the knowledge of an inherent bias.  Think WBD Iraq, where a small footnote in a single intelligence report led to second and third order effects that allowed the piece of information to gain more ground then it should—it created an echo chamber.

Before applying this to intelligence reporting, we decided to use a citation network to demonstrate capability.  We took the first 200 sources, sorted by relevance, using the keyword “network science.”  Then we built two edge list: First, we built a source edge list that has an “ID” column and a column with the source name.  Second, we built a citation edge list that has an “ID” column and a column with each sources citations.  So, we have list of 200 sources and about 5,000 citations.

Our code merges the data set so that each source has each of its citations with the correct ID, and we plot it.  However, we are trying to get it to plot as a DAG.  In our pseudocode, we have tried this twice unsuccessful.  We plan to do transitive reduction after we can make a DAG to get at our point of a metric of the echo effect.

Two Questions:
1)  Did we merge the data correctly or should we do more on the front end?
2)  We need to work on how to plot a DAG and how we can perform transitive reduction.

# Reading the Data into R
cit.citations <- read.csv(file.choose())
cit.sources <- read.csv(file.choose())

# Merge the Data Frames
cit.total <- merge(cit.sources, cit.citations, by="ID")

# List all the sources
sources.vertices <- cit.total$Sources
sources <- (as.character(sources.vertices))

citation.edges <- cit.total$Citations
citations <- (as.character(citation.edges))

total <- rbind(sources,citations)

# Redefining the Data Frame into a Network Object
net.cit.obj <- graph_from_data_frame(d = cit.total, directed=TRUE)

# Network - using dplyr package
cit.test <- dplyr::inner_join(cit.sources,cit.citations, by = "ID")[,-2]
cit.test.network <- as.network(cit.test, directed = TRUE, loops = FALSE)
as.sociomatrix(cit.test.network)
as.edgelist(cit.test.network)
plot(cit.test.network)

  Network with merged data


texts: For text match problem, what is the different between question-question match and question-answer match?

texts:
I know question-question match is a text similarity problem.

What about question-answer or question-doc match? It is used in information retrieval.

question-question match is indeed text similarity. But how do you define question-answer similarity?

Thank you!!


texts: algorithm to auto-download articles from the internet

texts:
I have an issued in my homeworks and I thinked if there is an rxisted algorithm or if can i create new one that takes key words like "germany" and "polution" and parses in google scholar. It parses fpr example the 10 first reults and each time it finds the key words in a specific part from the article ( just in the introduction) it downloads it.

Any one can help me ith any infrmation that can help me in this idea? Is it possible? what kind of alforithms should it be? ...


texts: Creating a Object Detection model from scratch using Keras

texts:
I have a dataset containing 330 images which contain guns. Along with the images, I have a text file associated with each image file which contains,

The number of objects ( guns ) in the image.
Coordinates for bounding boxes around the gun in the image.

I need to train a model which takes an image as an input and outputs 4 integer values which are the coordinates for the bounding box ( vertices of the bounding box ).

For training an object detection model, should the image be kept as an input and the coordinates as the output of the model? Should there be Convolution layers for feature extraction and then FC layers for learning the features for producing 4 outputs ( coordinates of the bounding box )?
Is this notion of the model architecture correct? Any other tips/suggestions?

I am creating this model entirely in TensorFlow Keras without using any of the pretrained stuff.


texts: What is the meaning of term Variance in Machine Learning Model?

texts:
I am familiar with terms high bias and high variance and their effect on the model.

Basically your model has high variance when it is too complex and sensitive too even outliers.

But recently I was asked the meaning of term Variance in machine learning model in one of the interview?

I would like to know what exactly Variance means in ML Model and how does it get introduce in your model? I would really appreciate if someone could explain this with an example.


texts: What courses / subjects are most important to the field of Data Science?

texts:
I've taken it upon myself to begin a career change. I have a decent background in mathematics, but lack in programming or data science specific skills (such as data munging). I have been looking through data science curricula and already feel overwhelmed by the amount of subjects a proper data science curriculum contains. Just to name a few: natural language processing, machine learning, R, Python, SQL, NoSQL, probability, statistics, numerical methods, algorithms, and the list goes on.

I can't afford to go back to school, but if my goal is to get into data science, what subjects should I focus my attention on?

I fully understand "data science" is a very big field, but certainly there must be several subjects that are important to all, if not the vast majority, of these sub-fields. I suppose this is essentially what my question is asking.  I have not yet obtained the amount of exposure to determine what I want to specialize in, so at this point, my concern is obtaining a foundation that wouldn't restrict me from entering any particular sub-field, or in other words, studying the subjects that benefit all sub-fields.

Note: Math isn't as much a concern for me compared to the other subjects.


texts: Eliminate input in gradient by clever choosing of cost function in neural networks

texts:
In http://neuralnetworksanddeeplearning.com/chap3.html

The author explains that for for a single neuron in neural net, by choosing cost function as cross entropy we can eliminate the derivative of activation function in the gradient term, if the activation function is choose to be sigmoid function.

In the problem, he asks why we cannot eliminate the input to neuron term x in gradient of cost function with respect to weights.

I had the following reasoning, in order to compute gradient of Cost, we use chain rule, and derive cost with respect to activation, and activation with respect to  (w * x + b), and the sum with weight.

For a weight Wi, the derivative of (Summation w * x + b) w.r.to Wi is always xi, and the derivative of activation function cannot know that, so it can never eliminate Xi unless its zero.

Or is there any other subtle reasoning?


texts: Classification of multidimensional data to multidimensional clusters with a varying subcluster structure

texts:
I have a large dataset with mixed (numerical, categorical, textual) data that I need to classify. The clusters are well-defined, but multidimensional (i.e. vector-valued) and have a varying structure as follows:

Cluster Structure 1: $(x_1, * , * , ... , * )$

Cluster Structure 2: $(x_1, x_2, * , ... , * )$

Cluster Structure 3: $(x_1, * , x_3, ..., * )$

...

Cluster Structure N: $(x_1, x_2, x_3, ... , x_m)$

where asterisks ($*$) indicate missing component entries, and each component, when present, (e.g. $x_2$) is a cluster to be identified. The Cluster Structures are disjoint in the sense that each member of the data set is only assigned to one Cluster Structure, but Cluster Structures can share components. For example, Cluster Structure 1 only consists of one "subcluster" ($x_1$) to be identified, whereas Cluster Structures 2 and 3 have two subclusters to be identified, with one of the clustering subproblems ($x_1$) shared and the other ($x_2$ vs. $x_3$) being different. The subcluster $x_1$ (and only that) is present in all Cluster Structures.
For each member of the dataset, I need to classify both the Cluster Structure and the associated tuplet of component clusters $x_i$, but I am not sure if it is better to treat these clustering problems sequentially (first structure then subclusters) or independently of each other (separate classifier for each structure). I could, of course, also define a "metacluster" as $(S, x_1, x_2, x_3, ..., x_m)$ where $S$ is the Cluster Structure id to be classified, but then I'd be treating each cluster structure identically when in fact classification within Cluster Structure 1 (once identified) is potentially much easier than in Cluster Structure N as there is only one subcluster to be identified (namely $x_1$).
I would be interested in any existing solutions (algorithms, research papers etc) that would apply, even remotely, to this problem.


texts: Performance metrics changing significantly based on batch size

texts:
I am working on a binary classification problem where there is significant class imbalance (minority class makes up nearly 10%). The dataset has ~15,000 observations and I have split this in to a training, validation and test set (that are stratified).
Using PyTorch I build a neural network with 5 fully connected layers (using ReLU activation), CrossEntropyLoss and SGD optimiser. Below is parts of my code
The problem is that my training vs validation loss changes a lot based on batch size (passed in the DataLoader). If I use a batch size of 64, the loss functions look like

which is quite odd. But if I use an unconventionally large batch size of say 1000, it looks like:
This looks more familiar but I can't make sense of what is going wrong here. I am also seeing that the training set reaches a high recall fairly quickly (after ~4 epochs) while the validation set improves slowly. So there seems to be an issue of overfitting as well.
I don't really know where I am going wrong: my neural network architecture consists of 5 fully connected layers with appropriate input and output dimensions. I initialise the weights. The forward function applies ReLU to the inputs (I don't use Softmax because I only need to classify 0 or 1 so I thought I can simply use argmax, see the 'c' variable in the code above).
I have tried setting Shuffle to true in the training_loader but this produces highly fluctuating training loss values.


texts: Separation of two probabilistic distribution: gaussian and exponential

texts:
I have a dataset where I highly suspect some correlation by two variables, based on my understanding of the problem. I would have a linear correlation (sort of SVD decomposition or a plane such as $y=mx+p$ for the main axis separation) of these variables. Assuming that I find the right separation, I would expect the principal axis to follow a gaussian distribution (high noise) and the secondary axis to follow and decreasing exponential probability (or maybe a poisson distribution with $\lambda$ close to 1).
My question is: how to test that and how would I extract the key parameters ($\lambda$ for exponential distribution, $\mu$ and $\sigma$ for the gaussian distribution and more importantly, the $m$ and $p$ for the $y=mx+p$ plane separation?)


texts: Deriving Feature Map formula for Inhomogeneous Polynomial Kernel

texts:
How do we derive/extrapolate the feature map for inhomogeneous polynomial kernels for degree $d$, define as $$K(x, x^\prime) = \bigg( (x \cdot x^\prime) + c\bigg)^d$$

I know we don't need to know a valid feature map to use the inhomogeneous polynomial kernels, I'm simply curious as to how one would derive/extrapolate it.


texts: Probability Calibration : For 2D image data, how to use the calibration?

texts:
I have a model which takes 2D input data and does multi class classification in keras. I would like to plot the probability calibration curve. However, using the scikit function, it returns an error saying that it cannot process 2D data. Please help.

My code is:

model= custom_model()
# define and fit calibration model
calibrated = CalibratedClassifierCV(model, method='sigmoid', cv=5)
calibrated.fit(x_train, y_train)

The error is:

Traceback (most recent call last):

   File "<ipython-input-41-bcfb033e1749>", line 1, in <module>
     yhat_calibrated = calibrated(x_train, x_test, y_train)

   File "<ipython-input-39-a3023fc5b92f>", line 100, in calibrated
     calibrated.fit(x_train, y_train)

   File
 "/home/eee/anaconda3/lib/python3.6/site-packages/sklearn/calibration.py",
 line 133, in fit
     force_all_finite=False)

   File
 "/home/eee/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py",
 line 756, in check_X_y
     estimator=estimator)

   File
 "/home/eee/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py",
 line 570, in check_array
     % (array.ndim, estimator_name))

 ValueError: Found array with dim 3. Estimator expected <= 2.

If not scikit, are there any other libraries that can help?


texts: How to represent the number of neurons in an LSTM for architecture schematic?

texts:
I'm trying to visualise a neural network schematic and found a great tool for building schematics here http://alexlenail.me/NN-SVG/index.html. I've edited the SVG file to change one of the dense layers into a LSTM layer, and the input to time series instead of singular neurons.

At the bottom of the image there is some set notation detailing how many neurons is in each layer. I'm not too familiar with set notation. I'm not quite sure how to represent the LSTM layers the number of neurons in accordance with the set notation given?

Its a singular LSTM layer with hidden_size=8 and the input/output size is obviously just the previous/next layers (5 and 8).

Also, the same question applies for the initial time series inputs; how to represent 5 time series of (say 100 points long) in set notation?

Any additional advice on how it might be better to represent the architecture would also be appreciated! Thanks!


texts: Trying to come up with a feature to improve emotion classifier based on facial movement using facial landmarks

texts:
I managed to create an emotion recognition system that uses dense optical flow on each entire frame. While the accuracy range is within 80-90% with cross-validation, I am aiming to improve the accuracy of the program.

There are four emotions: Neutral, happy, surprised, and angry. So far my classifier works pretty well, though it tends to over guess 'neutral' when  the answer is 'happy' or 'surprised'. This tends to happen when the mouth is only slightly opened, but the subject is visible smiling or have mouth opened in shock while the classifier still thinks the mouth is closed.

Confusion Matrix for Dense Optical Flow:
[[27 22  0  0]
 [ 0 57  1  0]
 [ 0 12 60  0]
 [ 0  9  3 68]]
Accuracy: 80-90% range

There is something I want to try in order to solve this though.

I have the ability to get the position of facial landmarks, though I don't know what I can do to turn this information into an effective additional feature I can use that would increase the accuracy. I was thinking of just simply getting the face landmark coordinates at the end of each video, but I feel like that it would not be the solution to differentiate between a closed mouth and slightly opened one (the difference in coordinate values will be small I think and am guessing that machine learning won't notice the difference).

I considered the possibility of just simply taking a still image of the subject's mouth and just analyzing that, but rejected it as it vulnerable to factors like lighting and people's appearance, and inconsistent matrix sizes. Plus I want my additional feature to take advantage of facial movement tracking.

I was wondering if there is a smart way to implement facial landmark tracking into a feature that would increase the accuracy of my classifier by dealing with my classifier's tendency to over predict the emotion 'neutral'. Any ways I can accomplish that?


texts:
Hi I have fairly very short time series data. The data set has number of systems $s_{1}, s_{2}, s_{3},..s_{n}$. For each $s_{i}$ we have recorded number of failures on each day. As of now, we have recorded 30 days. I would like to know can I use LSTM for predicting number of system failures for next day.
How about using Vector Auto Regression?. Any starting pointers and code references will be useful. Thank you.


texts: What is the shape of the vector after it passes through the TfidfVecorizer fit_transform() method?

texts:
I am trying to understand what happens inside the IDF part of the TFIDF vectorizer.

The official sci-kit Learn page says that the shape is (4,9) for a corpus of 4 documents having 9 unique features.
I get the Term Frequency (TF) part, it makes sense to me that ( for every unique feature(9), for each document(4) we calculate each term's frequency, so we get a matrix of shape (4,9)
But what doesn't make sense to me is the IDF part the formula for IDF is

So applying this formula, for every feature (9)  we calculate the log ( total no.of documents / no.of docs having the term or feature in it)
I think
This will result in a shape of (1,9) , please correct my understanding here.


texts:
I have list of log comments in CSV file. I want to cluster those log comments using K-Means and after that I want convert each cluster comments into general form.
for eg. I have bunch of comments in one cluster which starts from "Reservation number failed......" and I want to convert those comments into particular comment like "Reservation failure".

I can achieve this by giving specific name to each cluster after seeing each cluster. But I don't want like this. I want to create intelligent model which automatically create generalized comments for me.

I would not like to assign name to each cluster. Basically I am done with clustering part. that is, I have lets say 3 clusters as below

cluster 0 : list of comments like "Reservation number failed......",
total comments: 15
cluster 1 : list of comments like "Request timeout failed due to
......", total comments:9
cluster 2 : list of comments like "Dinning reservation successfully
completed...", total comments: 5

I want to build model that intelligently assign name to each cluster by its contents. for eg .

cluster 0 will get name as "Reservation failure"
cluster 1 will get name as "Request timeout failure"
cluster 2 will get name as "Dinning reservation successful"

if after training more data with some different comments. it should create another cluster and assign the name as per content.


texts: Using context to improve object detection and image classification

texts:
we are building an image classifier using the Tensorflow Object Detection API.

Our images often appear in groups, e.g. a cluster of dogs playing in the grass. Ofter our model will recognize some of the dogs - but not all of them. This is obviously a mistake a human would never make.

I am researching whether there are any examples of object detectors / image classifiers using context to improve results? I.e. knowing that there are three dogs nearby would increase the model's propensity to find a fourth?

Another way to potentially attack this problem would be to identify clusters of objects -> then reexamine the cluster only to identify the number of objects within the cluster. The complexity with this issue is that some of the objects we are examining appear in clusters, while others do not.

So to summarize, my two questions are:

Is there research on, and how would you suggest I go about improving an object detector by using contextual features within an image?
In situations where there are clusters of objects, would you recommend recognizing those clusters as individual images and then subsequently processing them to identify the number of images within the cluster? Are there examples / is there research on this?

Thanks in advance - obviously doing my own research as well, but keen to hear if the community has any thoughts/examples!


texts: Optimizing Tensorflow Graph Trace for Ram

texts:
I'm implementing an attention model in Tensorflow 2.0.   One 'issue' I've run into is that building the graph for a very long decoder can use a lot of ram and take a long time to compute.  That is, if your step function is something such as:
@tf.function
def train_step(inp, targ, enc_hidden):
  loss = 0
  with tf.GradientTape() as tape:

    enc_output, dec_hidden = encoder(inp, enc_hidden)

    for t in range(1, targ.shape[1]):
      # passing enc_output to the decoder
      predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)

      loss += my_loss_fun(targ[:, t], predictions)

      # using teacher forcing
      dec_input = tf.expand_dims(targ[:, t], 1)

  batch_loss = (loss / int(targ.shape[1]))

  variables = encoder.trainable_variables + decoder.trainable_variables

  gradients = tape.gradient(loss, variables)

  optimizer.apply_gradients(zip(gradients, variables))

  return batch_loss

Which is basically right from the tutorials.  If the target sequence length (i.e., the query sequence) is very long, the graph will be very memory intensive to build and take a long time.  After the trace it actually trains quite well, so that's not an issue.  I assume this is because Tensorflow is tracing the graph the first time and with a loop like this it has to build a very large graph (especially when the encoder and decoder is full of layers and attention mechanisms).  Actually, when debugging, it appears to trace through twice before running the graph.  Are there ways to mitigate this so that very large models can be loaded into memory, or is this just a consequence of training a model in a loop like this?


texts: What's the of all values above some percentile called? How do I get it in pandas?

texts:
Let's say I have a list of numbers, and I want the mean of all numbers that are greater than the 95th percentile. Is there some standard term for that value? ("Mean of a histogram bin"? "Conditional mean" on something?)
Is there a pandas standard library function to compute it?


texts: Do learning rate scheduler have any significant improvement or redundant on Adam optimizer?

texts:
As in paper, Adam optimizer is adaptive learning rates algorithm.
Is learning rate scheduler become redundant when use with Adam and AdamW ?
Is it best practices to use learning rate scheduler with Adam/AdamW ?


texts: K-fold-cross-validation if training dataset is much smaller than test dataset?

texts:
I'm a beginner in machine learning and I have a special case in which I have only a small training dataset of about 500 images and a test dataset of 10,000 images.
Does it still make sense to do a 10-fold-cross-validation or Repeated-Cross-validation on the training data? Or would this be not necessary anymore due to the large test dataset?

Many thanks in advance


texts: Correct order of operations involved into Dropout

texts:
Suppose we have CNN with any hidden layer with activation followed by dropout layer. What is the correct precedence of activation and dropout operation if dropout implementation is inverted dropout and CNN mode is training mode? Do I need to compute activation in the first layer and then apply dropout with division by retain probability p, or I need to apply activation to the result of the division?

Say we have the following keras code

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

If I understand correctly, dropout with division by p will be applied to the activated result:

result = [survive_mask] * relu(output)/p

Is this correct? Wouldn't it be more natural to have

result = [survive_mask] * relu(output/p)

because otherwise dropout operation breaks activation value normalization (i.e. to [0, 1]) ?


texts: Detecting cats visually by means of anomaly detection

texts:
I have a hobby project which I am contemplating committing to as a way of increasing my so far limited experience of machine learning. I have taken and completed the Coursera MOOC on the topic. My question is with regards to the feasibility of the project.

The task is the following:

Neighboring cats are from time to time visiting my garden, which I dislike since they tend to defecate on my lawn. I would like to have a warning system that alerts me when there's a cat present so that I may go chase it off using my super soaker. For simplicity's sake, say that I only care about a cat with black and white coloring.

I have setup a raspberry pi with camera module that can capture video and/or pictures of a part of the garden.

Sample image:

My first idea was to train a classifier to identify cat or cat-like objects, but after realizing that I will be unable to obtain a large enough number of positive samples, I have abandoned that in favor of anomaly detection.

I estimate that if I captured a photo every second of the day, I would end up with maybe five photos containing cats (out of about 60,000 with sunlight) per day.

Is this feasible using anomaly detection? If so, what features would you suggest? My ideas so far would be to simply count the number of pixels with that has certain colors; do some kind of blob detection/image segmenting (which I do not know how do to, and would thus like to avoid) and perform the same color analysis on them.


texts: Why is patsy used to prepare data for logistic regression?

texts:
I'm pretty new to both ML & scikit-learn. I've noticed that some example tutorials & codes online use patsy's dmatrices to prepare data for logistic regression. I don't understand why this is done. Example
In the case above for instance, isn't it sufficient to use the data in the dataset directly to train the logistic regression model? What exactly is the point of using patsy?
Thanks!


texts: Can somebody please give me some reference on implementing SVM using PySpark.


texts:
In Q-learning, how to tell the agent that action $a_7$  is unavailable from within state $s_{t}$?

Is supplying a very large negative reward good, or might throw it off-track?
From what I read (link), it's best to work with small rewards (between -1.0 and 1.0)


texts: Does random forest re-use features at each node when generating a decision tree?

texts: Learning about random forest for the first time and I'm not clear if at each level, if we've already used a feature to make a decision, whether we can use the same feature again at further levels.


texts:
I have a non-function (not in closed form) that takes in a few parameters (about 20) and returns a real value. A few of these parameters are discrete while others are continuous. Some of these parameters can only be chosen from a finite space of values.

Since I don't have the function in closed form, I cannot use any gradient based methods. However, the discrete nature and the boxed constraints on a few of those parameters restrict even the number of derivative free optimization techniques at my disposal. I am wondering what are the options in terms of optimization methods that I can use.


texts: Train an RNN with image sequences of varying length in keras for regression

texts:
How to program a RNN model with image sequences of varying length in keras for regression? (Just assume the output is some predefined continuous values)
I've read up things about training RNN on sequences with varying lengths like: here and here. But they all refers to numerical sequences. How can I deal with it in terms of image sequence of varying length in keras? Let's say I have sequences of varying length where each image sized 9*16 pixels. How to implement this in Keras?
Note: I also need the model to output intermediate values in hidden state.
Thanks in advance.


texts: What approaches could be used to teach a model to make appropriate brush strokes to reproduce a painting?

texts:
I'm pretty new to ML and most of the supervised techniques I've discovered so far are along the lines of

Pass model training set inputs and get it to generate predictions
evaluate predictions (predicted 'y') against real results (actual 'y') for the training set
Perform some kind of gradient descent to tweak the model parameters to give outputs that are closer to the real results

The precise method of training can vary, but the common theme in the examples I've seen so far is that the outputs of the model are the actual things that we want to get out of the system. For example, we might be trying to predict house prices - in this case, an output of the model would actually be a predicted price, and it's that that I would care about as part of my cost function and as a user of the system.

I'm imagining a scenario where we have a robot that's able to hold and use a palette and paintbrush, and we want it to copy existing paintings. (Strictly theoretical - I'm not trying to forge anything here!).

What I imagine I'd want is to

present robot with example input (existing paintings)
have the robot run a model where the output is some brush motions
evaluate the painting resulting from the brush strokes against the original painting

The difference here is that the output of my model is 'brush strokes', but that output (from the model) isn't ultimately what I care about - it's what's rendered by those brush strokes. I don't mind if the robot makes the same strokes that the original artist made, but I do mind about the similarity of what's rendered by those strokes to the original.

So I am wondering what areas of ML I need to look into to find something that works this way, where our cost or fitness is evaluated based on some transform of the model output, rather than the model output itself. I tried searching for examples of where a transform of the model output was incorporated into the cost function, but I couldn't see that it was a common technique.


texts: after cross-validation, what measure of 'stopping' training should i use when training final model on whole dataset?

texts:
I have performed 10 fold cross validation for performance estimation on a keras model. I now want to train a final neural network using ALL data for the FINAL model that will be deployed.
Given that this is the recommendation (to generate final model on all data), and I now no longer have any holdout data, conceptually, could I also average epochs taken to train during cross validation over the 10 folds to determine an epoch range that will not result in overfitting? or should i simply run 'x' epochs, overshoot and then simply determine the plateu?
is there a general rule here?
or... should I still have a validation split during training but no test set. This option makes no sense to me as this is exactly what cross validation was for.
thanks!


texts: Calculate correlation between two sensors regarding the time

texts:
I am new in all this, so I am sorry if I am asking something stupid.
I have two-time series - the measurement of the value by two sensors every 5 min and I want to see are the measurements correlated in time (are the measurements similar). So far now I only found questions related to autocorrelation and classic correlation. My dataframe looks like this:

                  id      time                      sensor1      sensor2
                  1       24.1.2020. 00:00:00        0.052         0.631
                  1       24.1.2020. 00:05:00        0.812         0.102            ....
                  1       24.1.2020. 00:10:00        0.326         0.500             ....
                  1       24.1.2020. 00:15:00        1.021         0.999             ....
                  1       24.1.2020. 00:20:00        1.033         1.000             ....

and so on for 10 days.

So, time here is a really important aspect because measurements depend on part of the day, of climate condition during the day and so on. I saw on some question that people are suggesting pandas function:

corr = df['value1'].corr(df['value2'])

But as I see, this line doesn't include the time.
Also if you have some course for data science to recommend, it will be appreciated. I am already taking something.


texts: How to combine PCA and MCA on mixed data?

texts: Suppose I have mixed data and (python) code which is capable of doing PCA (principal component analysis) on continuous predictors and MCA (multiple correspondence analysis) on nominal predictors. Is it possible to combine results from PCA and MCA into one?


texts: How does testing a CNN work after training?

texts: i mean that how CNN know the output probability will lead to following class. i mean that if i train the CNN on the Dog and cat picture or data set so at the end of the know that this probability will lead to the Dog or cat...


texts: Python - Feature Selection - Should I remove bad variables?

texts:
I've this code to print the importance of each variable on my model:

importances = trained_model.feature_importances_
std = np.std([trained_model.feature_importances_ for trained_model in trained_model.estimators_], axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(training_features.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

I print a lot of variables with feature ranking as 0.0. Should I remove that variables? I can I do it using Python?

Like this:

df = df.drop('Col_A', 1) WHERE importances[indices[f]] = 0

Thanks!


texts: predict gives the same output value for every image (Keras)

texts:
I am trying to classify images and assign them label 1 or 0. (Skin cancer or not).

I am aware of the three main issues regarding having the same output in every input.

I did not split the set and I'm just trying to apply the CNN on the train set, I know it doesn't make sense but it's just to verify how it's working. (Predicting on the unlabeled data gives exact same probability)

I have verified the three main points:

1: Scaling the date (both image size and pixel intensity values)
2: Taking a low learning rate
3: I only tried with small epochs 6 at most because of the computation time, is it worth it to let it run one day just to see results with more epochs ?

Anyway I can't understand how a bad training could lead the network to give same class probability every time ?

I tried the on batch options etc.. doesn't change anything.

Accuracy is very low as this kind of classification is not really suited for CNNs but this shouldn't explain the weird result though.

Here is my output :

Here are different parts of the program:
Model :

Resizing :

Thanks for help and sorry for the ugly screenshots.


texts: I am currently working on a project and I need to do projected gradient descent instead of vanilla gradient descent on a network. I am unsure if current deep learning frameworks have that functionality. I tried searching online but didn't get much. Any help in this regard is appreciated!


texts: NCF Recommender- The target encoded within the model input, why doesn't it overfit easily?

texts:
In the recommender system NCF, the input is a batch of user-item interactions (one-hot encoded) and the output is a 0-1 score of whether the item has been bought or not:

This seems to indicate that the item input vector that the model is trained on already contains y. I understand the purpose of this, but doesn't that lead to a dangerously high chance of overfitting?


texts: how to vertical integration along specific levels

texts: I know it will sound silly but I have got a small question to ask. I want to do vertical integration for atmospheric data. Suppose say, my data contains dimensions as (time, level, long, lat). Now the level is from 1000hpa to 5hpa, and I want to integrate it from 800hpa to 550 hPa. So can anyone help me by saying how to do this, like whether is there any function in python of doing such integration?


texts: Does the network learn based on previous training or does it restart? Matlab, neuralnetworks

texts:
In Matlab, if you build a simple network and train it:

OP = feedforwardnet(5, 'traingdm');
inputsVals = [0,1,2,3,4];
targetVals = [3,2,5,1,9];
OP = train(OP,inputsVals,targetVals);

then you train it again so another OP = train(OP,inputsVals,targetVals);

What is happens to the network? Does it train again based on what it learned the first time you did OP = train(OP,inputsVals,targetVals); or does it train as if it were the first time training the network.


texts: How do we determine what is learnt by a ML model?

texts:
I am trying to investigate/justify the output of a random forest regression model for a financial problem, where justification of the output is also important. In that context, for the random forest regression model that I built, I'm using the tree interpreter for justification of the output using the contributions. But is there any other way I can do this better? Is this the best approach?

The idea is to be able to explain the output of the random forest regression model to a layman supported with technical points.


texts:
I am new to machine learning so please bare with me. I'll try to keep this short and sweet.
We are building a makeup simulation and recommendation system. My part is to recommend a makeup which is personalized to the user and also on par with the current makeup trends. I will be building a set of rules with the help of a beautician that will say which makeup is suitable for a particular set of features.
The outputs will be makeup for: foundation, lipstick, eye shadow
The input features are (tentative): skin tone, hair colour, dress colour, morning/evening, type of event attending, etc
So after I build this set of rules, I will be able to select the appropriate makeup to the given input. What I need is to be able to recommend makeup that is also trending amongst similar users because makeup styles are constantly evolving and I don't want to recommend the same makeup every time the same input features are given.
Also I want to be able to personalize the recommendation, i.e. if the user's history shows preference towards a nude makeup, I want the recommendation to be as close as possible to their preferences.
I'd appreciate any help regarding how I should proceed or what algorithms I should use or anything at all...!!


texts: How to select one record from multiple record for a subject during analysis?

texts:
I have a dataset where I am working on a binary classification. I have two classes of subjects. One is Outpatients and Other is Inpatients. (66:33 is the class proportion)

My objective is to identify the risk factors that influence hospital admission (Inpatients).

But the problem here is, I have my dataset like as below

1) Let's say we have a subject called "John". He has visited hospital 19 times based on my data duration from Jan 2001- Dec 2005. All of his 19 visits are outpatients.

2) Let's say we have another subject called "Jack". He has visited the hospital 34 times based on data duration from Jan 2001-Dec 2005. Out of 34 visits, he has been admitted as inpatient 18 times and rest 16 are outpatient visits.

So now my question is

1) Usually for analysis, we only see one record per subject/individual. Right? But now on what basis should I pick that one record?

Meaning, for John out of his 19 visits, which one should I pick?

Similarly for Jack, out of his 18 inpatient visits, which one should I pick?

I choose only one out of 18 from Jack because we don't need his outpatient info as we already have a separate group of outpatients and jack is considered for Inpatient class (because he has inpatient records too unlike John).

2) Is it really necessary to have only one record per person for analysis? Is there anyway to do this? Or is it like I have to represent in aggregate form the info of multiple records in one record? Is there any theory that allows analysis of multiple records for an individual?

Hope my question is clear and kindly request you to help me


texts: what predictive analysis will work with this data set?

texts:
I have a dataset that includes all of the building permits that were issued for homes within a city. I have the data in a "snapshot" from a year ago (and potentially a few others, older snapshots) and today.

Included in the data are things like home age and other attributes about the home. What I'm hoping to do is figure out what characteristics predict a future permit. For this, I was going to compare the year ago snapshot with the present snapshot to see which homes ended up getting permits (e.g., to remodel a bathroom) and determine if there's any correlation between characteristics that predict these.

My questions are the following:

What form would I need the data to do this? Assume I have all the characteristics in different "columns". Most of it will be categorical or binary (true/false). Dates will obviously be in date format. I have the ability to add additional flags in the past or current data that might indicate, for example, a new plumbing permit was issued.
What analysis would work best for this and any recommendations for tools/platforms to run this? Assuming hundreds of thousands of entries would be available to analyze.

Thanks.


texts: Which batch size to use when Batch Normalization?

texts:
I want to train a CNN in Keras (optimizer Adam) and by using batch normalization after every ConvLayer and before every activation layer.
So far I mostly see examples in which training is carried out with a batch size of 32 or 64 samples.
Should it not be ensured that the last batch in every epoch still contains 32 or 64 samples and not significantly less samples?
What do I mean, if I have 500 training samples for training, would a batch size of 50 not be better than a batch size with 32 oder 64 samples?


texts: Training a Variational Autoencoder (VAE) for Non-Uniform Random Number Generation

texts:
I have a complicated 20-dimensional non-uniform distribution and would like generate samples from it. I have considered training a VAE to do so, but my problems are the following:

Is my approach fundamentally or logically flawed? Specifically, because unlike image data, the random numbers are of geometric nature and thus take negative values and could also be considered noisy.
How do I find the right architecture aside from simple trial and error? Obviously, I do not necessarily need 2D-Convolutions. But instead, 1D-Convolutions could be considered a good choice to capture the correlations (i.e. modalities of the distribution). I'm also not sure about how I properly decide on the number of hidden layers and nodes.


texts:
If i wanted to do website categorization is there any prepared toolkit like pretrained model, or open data for categories with keywords?
For example google listed some categories here https://cloud.google.com/natural-language/docs/categories so to have for example data for each category together with associated keywords.


texts: Use machine learning to predict next schedule meeting for sales officers

texts:
I have a project with data of sales field officers who visit their customers and enter the progress details. Visit can be an order or any kind of customer interaction.

Let's say one sales guy has around 1000 customers, It's only natural that he might skip some customers which can result in loss overtime. So I have the data for the visits done by him since the customer was onboarded. What model should I use to check the old frequency of visits done by him and factors which are stated below so that I can say like:

"Do you want to get in touch with "this" customer. ?"

So basically by checking his previous interaction rate, In case if he forgot to visit, I need to recommend like that.

Data points are like:

date/time - date when visit done
remarks - what was discussed
there is a status which is related to internal
and some other customer details.

So what model should I use or which technique do you think is best suited for this problem. I'm fairly new in machine learning and kind of learning it by doing.


texts: Spark: How to run PCA parallelized? Only one thread used

texts:
I use pySpark and set my configuration like following:
spark = (SparkSession.builder.master("local[*]")
        .config("spark.driver.memory", "20g")
        .config("spark.executor.memory", "10g")
        .config("spark.driver.cores", "30")
        .config("spark.num.executors", "8")
        .config("spark.executor.cores", "4")
        .getOrCreate())
sc = spark.sparkContext

If I then run PCA:
from pyspark.ml.feature import PCA

pca = PCA(k=50, inputCol="features", outputCol="pcaFeatures")
model = pca.fit(train)

Only one thread is active and therefore the computation takes a long time.
How can I parallelize PCA in Spark?
I run on a local machine and did not configure a cluster in the configs.
Also I did not install the recommended ml packages, since the warning
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK

appears.


texts: Fuzzy Rules with more than two variable in python

texts:
I am trying to build a fuzzy inference system in python using skfuzzy library. I have 4 variables depending on which output class is decided.

def fuzzInferenceself():

### Input ###
    hf_very_poor = fuzz.trimf(hotel_facility, [0, 0.15, 0.3])
    hf_poor = fuzz.trimf(hotel_facility, [2.5,0.3,0.45])
    hf_average = fuzz.trimf(hotel_facility, [0.4, 0.5, 0.75])
    hf_good = fuzz.trimf(hotel_facility, [0.7, 0.85, 0.9])
    hf_very_good = fuzz.trimf(hotel_facility, [0.875, 0.92, 1.0])
    vc_less = fuzz.trimf(visited_count, [0, 0.05, 0.1])
    vc_average = fuzz.trimf(visited_count, [0.05, 0.2, 0.35])
    vc_many = fuzz.trapmf(visited_count, [0.3,0.45,0.55,0.7])
    vc_a_lot = fuzz.trapmf(visited_count, [0.65,0.8,0.9,1.0])
    rm_very_poor = fuzz.trimf(hotel_facility, [0, 0.15, 0.3])
    rm_poor = fuzz.trimf(hotel_facility, [2.5,0.3,0.45])
    rm_average = fuzz.trimf(hotel_facility, [0.4, 0.5, 0.75])
    rm_good = fuzz.trimf(hotel_facility, [0.7, 0.8, 0.9])
    rm_very_good = fuzz.trimf(hotel_facility, [0.85, 0.9,1.0])

## output ####
    class_very_poor = fuzz.gaussmf(class_score,1,0.5)
    class_poor = fuzz.gaussmf(class_score,1.75,0.65)
    class_average = fuzz.gaussmf(class_score,2.25,0.75)
    class_good = fuzz.gaussmf(class_score,3,0.25)
    class_very_good = fuzz.gaussmf(class_score, 3.5, 0.5)

def hotelFaclilityClassification(self,A):
    hf_vp= fuzz.interp_membership(hotel_facility, hf_very_poor, A)
    hf_p= fuzz.interp_membership(hotel_facility, hf_poor, A)
    hf_av= fuzz.interp_membership(hotel_facility, hf_average, A)
    hf_gd= fuzz.interp_membership(hotel_facility, hf_good, A)
    hf_vg= fuzz.interp_membership(hotel_facility, hf_very_good, A)
    return dict(hfVP = hf_vp, hfP = hf_p, hfAV = hf_av,hGD = hf_gd, hVG = hf_vg)

def visitCountClassification(B):
    vc_l = fuzz.interp_membership(visited_count,vc_less)
    vc_av = fuzz.interp_membership(visited_count,vc_average)
    vc_mn = fuzz.interp_membership(visited_count,vc_many)
    vc_al = fuzz.interp_membership(visited_count,vc_a_lot)
    return dict(vcL = vc_l, vcAV=vc_av, vcMN = vc_mn, vcAL = vc_al )

def roomFacilityClassification(C):
    rm_vp = fuzz.interp_membership(room_facility,rm_very_poor)
    rm_p =  fuzz.interp_membership(room_facility,rm_poor)
    rm_av = fuzz.interp_membership(room_facility,rm_average)
    rm_gd = fuzz.interp_membership(room_facility,rm_good)
    rm_vg = fuzz.interp_membership(room_facility,rm_very_good)
    return dict(rmVP = rm_vp, rmP = rm_p, rmAV = rm_av, rmGD = rm_gd, rmVG = rm_vg)

A similar function for price is: def priceClassification(D).

The rules are as follows:

"If Hotel facility score is Very good, visited count is alot, room facility is very good,prices is less then class is very good."

I do not understand how to code the rules. All sources I have seen takes one input and one output variable. But this is not the case in my code.

Can anyone give me a good resource or idea about how to code this rule?


texts: Hosmer Lemeshow Test, variable lengths differ

texts:
Doing a logistic regression in R version 3.1.2 and I want to conduct a Hosmer Lemeshow test

model <- glm(y ~ a + b + c + d, data = data, family = binomial)

summary(model)

Call:
glm(formula = y ~ a + b + c + d, family = binomial,
    data = data)

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-2.5607  -0.7642  -0.4456   0.8998   2.3812

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -7.827805   1.169216  -6.695 2.16e-11

a            0.582214   0.134207   4.338 1.44e-05

b           -0.697126   0.462983  -1.506  0.13214

c            0.027485   0.009669   2.843  0.00448

d            0.996241   0.163006   6.112 9.86e-10

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 504.53  on 373  degrees of freedom
Residual deviance: 381.91  on 369  degrees of freedom
  (3 observations deleted due to missingness)
AIC: 391.91

Number of Fisher Scoring iterations: 5

Using the function hoslem.test from package "Resource Selection"

hoslem.test(model$y, fitted(model), g = 10)

I get the following error

Error in model.frame.default(formula = cbind(y0 = 1 - y, y1 = y) ~ cutyhat) :
  variable lengths differ (found for 'cutyhat')

Looking at the code in the hoslem.test function, cutyhat is derived only from fitted(model) values, which I've verified contain no nas and are all integers, so why would they differ?

Pls help.


texts: how to create new columns in pandas using some rows of existing columns?

texts:
i have a dataset like this

my desire format is like this

I tried using index slicing eg
dll.loc[:4,'category'] = "CAPITAL FUND"
dll.loc[5:10,'category'] = "BORROWING"
but this idea is risky so is there any idea to solve this?


texts: How to create clusters of position data?

texts:
I am asking this question because the previous one wasn't very helpful and I asked about a different solution for the same problem.

The Problem

I have lateral positions, xcoord, of vehicles over time which were recorded as the distances from the right edge of the road. This can be seen for one vehicle in the following plot:

Each point on the plot represents the position of the front center of the vehicle. When the vehicle changes the lane (lane numbers not shown) there is a drastic change in the position as seen after the 'Start of Lane Change' on the plot.
The data behind this plot are like below:

  Vehicle.ID Frame.ID   xcoord Lane
1          2       13 16.46700    2
2          2       14 16.44669    2
3          2       15 16.42600    2
4          2       16 16.40540    2
5          2       17 16.38486    2
6          2       18 16.36433    2

I want to identify the start and end data points of a lane change by clustering the data as shown in the plot. The data points in the plot circled in red are more similar to each other because the variation between them is smaller compared to the data points in the middle which see large variation in position (xcoord).
My questions are: Is it possible to apply any clustering technique to segment these data so that I could identify the start and end point of a lane change? If yes, which technique would be most suitable?
I use R. I have tried Hierarchical clustering before but don't know how to apply it in this context. Please help.


texts: Stop CNN model at high accuracy and low loss rate?

texts:
I train my CNN model with a large number of epochs, with each epoch I print the training loss and accuracy, but there is a lot of high and low in these two metrics, I want to do early stopping with for example loss at 0.2 and accuracy at %95 or more because I get this at more than one epoch, my question is:

1- is early stopping done on train set or validation set?

2- if on validation set, should I print validation loss and accuracy for each epoch with with a train set loss and accuracy?

3- can you give an idea to do early stopping with a piece of code but not in keras, because I use tensorflow not keras?


texts: Keras categorical_crossentropy loss (and accuracy)

texts:
When training a neural network with keras for the categorical_crossentropy loss, how exactly is the loss defined? I expect it to be the average over all samples of
$$\textstyle\text{loss}(p^\text{true}, p^\text{predict}) = -\sum_i p_i^\text{true} \log p_i^\text{predict}$$
but couldn't find a definitive answer in the docs nor in the code. An authoritative reference is desirable.

Looking at the code I'm not sure if the computation is delegated to tensorflow/theano.

(There is an analogous question concerning the accuracy; the code is clearer, but I don't see a call to mean()?)

PS.
From this code, it appears that loss and accuracy are computed as in loss_and_acc(...) but before the last training epoch (keras version 2.0.4, same results for tensorflow and theano backend).

#!/usr/bin/python3

import numpy as np
from numpy.random import randint, seed

from keras import __version__ as keras_version
from keras.models import Sequential
from keras.layers import Dense

N = 4 # Classes
S = 10 # Samples

nn = Sequential()
nn.add(Dense(input_dim=1, units=N, kernel_initializer='normal', activation='softmax'))
nn.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

seed(7)
X = np.random.random((S, 1))
Y = np.vstack([np.eye(1, N, k=randint(0, N)) for _ in range(S)])
#for (x, y) in zip(X, Y) : print(x, y)

def loss_and_acc(NN, X, Y) :
    loss = []
    acc  = []
    for (p, q) in zip(Y, NN.predict(X)) :
        loss += [ -sum(a*np.log(b) for (a, b) in zip(p, q) if (b != 0)) ]
        acc  += [ np.argmax(p) == np.argmax(q) ]
    return (np.mean(loss), np.mean(acc))

print("Keras version: ", keras_version)

for _ in range(10) :
    print("Before:  loss = {}, acc = {}".format(*loss_and_acc(nn, X, Y)))
    H = nn.fit(X, Y, epochs=1, verbose=0).history
    print("History: loss = {}, acc = {}".format(H['loss'][-1], H['acc'][-1]))

Output:

Using Theano backend.
Keras version:  2.0.4
Before:  loss = 1.3843669414520263, acc = 0.2
History: loss = 1.3843669891357422, acc = 0.20000000298023224
Before:  loss = 1.3834303855895995, acc = 0.2
History: loss = 1.3834303617477417, acc = 0.20000000298023224
Before:  loss = 1.3824962615966796, acc = 0.3
History: loss = 1.3824962377548218, acc = 0.30000001192092896
Before:  loss = 1.381564486026764, acc = 0.3
History: loss = 1.3815644979476929, acc = 0.30000001192092896
Before:  loss = 1.380635154247284, acc = 0.3
History: loss = 1.380635142326355, acc = 0.30000001192092896
Before:  loss = 1.3797082901000977, acc = 0.3
History: loss = 1.3797082901000977, acc = 0.30000001192092896
Before:  loss = 1.378783941268921, acc = 0.2
History: loss = 1.378783941268921, acc = 0.20000000298023224
Before:  loss = 1.3778621554374695, acc = 0.2
History: loss = 1.3778622150421143, acc = 0.20000000298023224
Before:  loss = 1.3769428968429565, acc = 0.2
History: loss = 1.3769428730010986, acc = 0.20000000298023224
Before:  loss = 1.3760262489318849, acc = 0.3
History: loss = 1.3760262727737427, acc = 0.30000001192092896


texts: How does one fine-tune parameters and weights at the same time?

texts:
I have been having my hands full with training a model to classify web pages. This is the first time ever that I am doing this, so I know very little about ML. I'm here to learn. :-)

Currently, I am using a Pipeline with FeatureUnion to run a a GridSearchCV to tune my parameters. However, I am slightly confused about how this process ought to go. For the whole process of creating a predictive model I read this answer on "Machine Learning Steps" question, which was helpful, but I still have some questions.

First of all: are parameters independent of your model? The answer linked above suggests first testing different models, and then do parameter tuning. However, I started with tuning my parameters on a Naive Bayes multinomial model, with the idea to test the models (e.g. svm, kNN, NBmn) afterwards with the 'optimal' parameters. But the answer above seems to suggest otherwise. Does this mean that the parameter tuning depends on the best model you choose, which makes it useless to first tune the parameters?

Apart from that, I am also not sure how to correctly tune my parameters per feature. I have 13 features, and put them all in the Pipeline. For things like TfidfVectorizer for eight features, I'd like to tune max_features, ngram_range, and max_df. However, because this takes quite a long time to run I thought I'd start with two features, make sure I get the optimal value, set them in the Pipeline (not in the parameter tuning), and add a feature. However, I feel that this is not the way at all. Is it required to the complete tuning as a single step, or is it possible to do it in steps?

Lastly, I noticed that you can also pass the feature weights of FeatureUnion as a parameter, and tune it. But as far as I have found you can only add specific values to test (example below) and not a range (e.g. 0-1 with steps of 0.1), which makes it hard to set the parameter values as you've got to guess which could be a good value. Is there a better way?

parameters = {
    'union__transformer_weights': [
        {'f1':1, 'f2':0.4, 'f3': 0.2,'f4': 1,'f5': 0.4,'f6': 0.1},
        {'f1':0.4, 'f2':0.9, 'f3': 0.2,'f4': 0.4,'f5': 0.4,'f6': 0.2},
        {'f1':0.8, 'f2':0.7, 'f3': 0.2,'f4': 0.7,'f5': 0.7,'f6': 0.4}
    ]
}


texts: Correlation/distance between sparse vectors

texts:
I am looking for a metric for comparing gene count tables. These are long columns of data (a few millions genes by a few dozen samples), with all non-negative entries, about 90% of which are zeros. The goal is to compare the performance of several tools/algorithms that these tables originate from, by comparing the resulting tables among themselves or with the expected counts (in a case of sumulates data). In principle, one compares on a sample-by-sample basis, but comparing different samples might be also of interest, e.g., to filter out spurious correlations.
What I am using now is Spearman rank coefficient, taking account for the fact that some entries have identical ranks (certainly the zeros). I am looking for an approach more adapted to this setting (and preferably robust to outliers) and will appreciate suggestions.


texts: How to run tensorflow model twice before computing the loss

texts:
I want to compute a loss function which uses output of the network twice on different inputs. For example as hypothetically,

first_output = model(first_input)
second_output = model(second_input)
loss = mean_absolute_error(first_output, second_output)

How to achieve this in tensorflow or keras?


texts:
I'm trying to classify a binary sample with Keras and I would like to classify as many correctly as possible, while ignore the ones where the model is not sure.

The fully connected Nerual network currencly achieves around 65% but I would like to get a higher result of correctly classified ones, while ignoring the ones where the model is uncertain.

Is there a way to tell Keras to simply ignore the ones where the model is uncertain and achieve a higher accuracy that way? Or is there a network design that could achieve this, for example feeding the result of the network striaght into a second part of it which then decides whether the prediction is likely accurate or not?

One way I was thinking of achieving this is by building a second neural network on top of it that decides based on the result of the first network and all the input data of it, whether the classification will be correct or not. Would that work, and if yes, is there no more elegant way of achieving this in one go, such as directly having the results feeding into a second part of the network that then decides if the prediction is likley accurate or not?


texts: How could I improve my FB Prophet forecast?

texts:
I've got 1325 days of revenue data and when plotting the components it makes 100% sense from a domain expert point of view, so the model is capturing the variations quite well (or it seems it does...). I've added the country holidays using m.add_country_holidays(country_name='GB')

However, when it comes to accuracy I'm getting the following averages:

MAPE: 0.3
MAE: 721,415

721,415 is not an acceptable error. Around 100K would be.

These are the MAE and MAPE plots:

Time-series plot:

Performance metrics (first 20):

What else can I do to improve the accuracy of this model? Thank you


texts: How to evaluate the clustering result when cluster numbers are not equal to data set class

texts:
When apply clustering algorithm with the multi-class data set and class numbers are not equal to the result cluster numbers(For example , When we use K-Means algorithm by setting K = 3 apply with "yeast" data set that has 14 classes).

What is the proper way to evaluate the result in this situation ?


texts: Is it possible to detect a person using facial recognition on the fly?

texts:
I am working on a small app for face detection in Python using face_recognition and opencv libraries.

I have trained the data on a number of people and used face recognition to login into the app.

If it detects the person from the picture, it allows him to log in, otherwise they are sent to the registration screen where the person has to upload his photos and other information.

So if a new registration comes in, I have to run the model again with additional data.

As the number of entries increases the model will take more time, affecting the time to identify a newly registered person.

How can I prepare for this slow-down, down the road?

Can I reduce the time taken to train the model again or train the model with incremental data?


texts: NLP Transformers: How to get a fixed sentences embedding vectors size?

texts:
I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences:

import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
camembert.eval()  # disable dropout (or leave in train mode to finetune)

def embed(sentence):
   tokens = camembert.encode(sentence)
   # Extract all layer's features (layer 0 is the embedding layer)
   all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
   embeddings = all_layers[0]
   return embeddings

# Here we see that the shape of the embedding vector depends on the number of tokens in the sentence

u = embed("Bonjour, ça va ?")
u.shape # torch.Size([1, 7, 768])
v = embed("Salut, comment vas-tu ?")
v.shape # torch.Size([1, 9, 768])

Imagine now, I want to calculate the cosine distance between the vectors (tensors in our case) u and v :

cos = torch.nn.CosineSimilarity(dim=0)
cos(u, v) #will throw an error since the shape of `u` is different from the shape of `v``

I'm asking what is the best method to use in order to always get the same embedding shape for a sentence regardless the count of tokens?

=> The first solution I'm thinking of is calculating the mean on axis=1 (mean embedding of tokens in the sentence) since axis=0 and axis=2 have always the same size :

cos = torch.nn.CosineSimilarity(dim=1) #dim becomes 1 now

u = u.mean(axis=1)
v = v.mean(axis=1)

cos(u, v).detach().numpy().item() # works now and gives 0.7269

But, I'm afraid that I'm hurting the embedding when calculating the mean!

=> The second solution is to pad shorter sentences out, that means:

giving a list of sentences to embed at a time (instead of embedding sentence by sentence)
look up for the sentence with the longest tokens and embed it, get its shape S
for the rest of sentences embed then pad zero to get the same shape S (the sentence has 0 in the rest of dimensions)

What are your thoughts? What technique would you use and why?


texts: Least squares with non-negative eigen values

texts:
I am trying to use least squares to solve a problem of the form

$u = - K v$

where u and v are vectors of size 3, and K is a 3X3 matrix. Where I want to estimate K, given u and v. I have multiple data for u and v, setup with the hope that the different data points are sufficiently linearly independent that a unique solution exists.

The way I have proceeded to do this is setup the problem as a traditional least-squares problem

$Ax = b$,

where b is now all the data I have for the 3 elements of u, and A has all the data for v, and K has been expanded into a vector x. Where x = (K11, K12, K13, K21, ... K33).

Due to some physical reasons I want to add an extra constrain. The constrain is that if I break the matrix K into a symmetric and antisymmetric part,

$K = S + A$,

then the eigenvalues of S should be positive, or atleast 2 of them be positive. In particular, S will have 3 eigenvalues, ($\lambda_1, \lambda_2,\lambda_3 $). In my problem $\lambda_1, \lambda_2>> \lambda _3$, and I would like $\lambda_1, \lambda_2 > 0$.

The physical reason is that $\lambda_1, \lambda_2$ are representative of diffusivities, and negative diffusivities are usually unjustifiable on physical grounds.

In my current solution, where I apply no constrains, the eigenvalues end up being positive almost everywhere (>80% times). I think the places where the values are turning out to be negative are regions where the data in u and v (or A and b) is not sufficiently linearly independent (any help on a good metric to check this would also be appreciated).


texts: Autoencoder features very different after each training

texts:
I trained a convolutional (3D) autoencoder on about 350 volumes. The reconstruction of the volumes by the decoder is really nice and there is nearly no difference between the input volumes (images) and the output.
My intention was to use the autoencoder for feature learning, extract the features after the encoder and use local sensitivity hashing (LSH) for a nearest neighbors search.

I now trained the autoencoder five times in a row using the same dataset split to gain information about the model sensitivity.
Iam aware of that there is always some randomness when training a model, nevertheless when applying the nearest neighbors search, each model finds totally different neighbors and it seems that each autoencoder learns totally different features.

The loss during training (MSE) varies between 2.5e-05 to 3.5e-05, which seems ok.

I would have expect the features of the models to be aproximately the same and I also would have expect to find the same neighbors.

What am I doing wrong and why are the features so different?

Thanks in advance for any advice,

cheers,

Michael


texts: Confused about how to apply KMeans on my a dataset with features extracted

texts:
I am trying to apply a basic use of the scikitlearn KMeans Clustering package, to create different clusters that I could use to identify a certain activity.  For example, in my dataset below, I have different usage events (0,...,11), and each event has the wattage used and the duration.

Based on the Wattage, Duration, and timeOfDay, I would like to cluster these into different groups to see if I can create clusters and hand-classify the individual activities of each cluster.

I was having trouble with the KMeans package because I think my values needed to be in integer form. And then, how would I plot the clusters on a scatter plot?  I know I need to put the original datapoints onto the plot, and then maybe I can separate them by color from the cluster?

km = KMeans(n_clusters = 5)
myFit = km.fit(activity_dataset)

       Wattage        time_stamp       timeOfDay   Duration (s)
    0    100      2015-02-24 10:00:00    Morning      30
    1    120      2015-02-24 11:00:00    Morning      27
    2    104      2015-02-24 12:00:00    Morning      25
    3    105      2015-02-24 13:00:00  Afternoon      15
    4    109      2015-02-24 14:00:00  Afternoon      35
    5    120      2015-02-24 15:00:00  Afternoon      49
    6    450      2015-02-24 16:00:00  Afternoon      120
    7    200      2015-02-24 17:00:00    Evening      145
    8    300      2015-02-24 18:00:00    Evening      65
    9    190      2015-02-24 19:00:00    Evening      35
    10   100      2015-02-24 20:00:00    Evening      45
    11   110      2015-02-24 21:00:00    Evening      100

Edit:
Here is the output from one of my runs of K-Means Clustering.  How do I interpret the means that are zero?  What does this mean in terms of the cluster and the math?

print (waterUsage[clmns].groupby(['clusters']).mean())
          water_volume   duration  timeOfDay_Afternoon  timeOfDay_Evening  \
clusters
0             0.119370   8.689516             0.000000           0.000000
1             0.164174  11.114241             0.474178           0.525822

          timeOfDay_Morning  outdoorTemp
clusters
0                       1.0   20.821613
1                       0.0   25.636901


texts:
So I am computing the training accuracy of my CNN on a batch training data and it almost always gives 100% accuracy. But I found from somewhere that in order to get the training accuracy, it is tested against the entire training data and not the specific batch that you used for your training.

Is training accuracy even computed? I wonder if you don't use batches for training. Then it will be 100% accuracy always?

I'm suspecting that this is due to using a small batch size.


texts: How to write a simple rule-based datetime range parser in python?

texts:
The dateparser package fails to detect texts like the following and generate a date range 'last 2 weeks of 2020': Should return 18th December 2020 - 31st December 2020 'first three quarters of 2018': Should return 1st January 2018 - 30th September 2018 'last 3 days of September 2020': Should return 28th September - 30th September 2020
Is there a simple rule-based parser in python that can detect the words last, first, 2, 3, 2020, 2018 etc. and give a date range in the datetime format? I do not want to rely on heavy packages like SUtime to achieve this.


texts: Entity Recognition in Stanford NLP using Python

texts:
I am using Stanford Core NLP using Python.I have taken the code from here.
Following is the code :

from stanfordcorenlp import StanfordCoreNLP
import logging
import json

class StanfordNLP:
def __init__(self, host='http://localhost', port=9000):
    self.nlp = StanfordCoreNLP(host, port=port,
                               timeout=30000 , quiet=True, logging_level=logging.DEBUG)
    self.props = {
        'annotators': 'tokenize,ssplit,pos,lemma,ner,parse,depparse,dcoref,relation,sentiment',
        'pipelineLanguage': 'en',
        'outputFormat': 'json'
    }

def word_tokenize(self, sentence):
    return self.nlp.word_tokenize(sentence)

def pos(self, sentence):
    return self.nlp.pos_tag(sentence)

def ner(self, sentence):
    return self.nlp.ner(sentence)

def parse(self, sentence):
    return self.nlp.parse(sentence)

def dependency_parse(self, sentence):
    return self.nlp.dependency_parse(sentence)

def annotate(self, sentence):
    return json.loads(self.nlp.annotate(sentence, properties=self.props))

@staticmethod
def tokens_to_dict(_tokens):
    tokens = defaultdict(dict)
    for token in _tokens:
        tokens[int(token['index'])] = {
            'word': token['word'],
            'lemma': token['lemma'],
            'pos': token['pos'],
            'ner': token['ner']
        }
    return tokens

if __name__ == '__main__':
sNLP = StanfordNLP()
text = r'China on Wednesday issued a $50-billion list of U.S. goods  including soybeans and small aircraft for possible tariff hikes in an escalating technology dispute with Washington that companies worry could set back the global economic recovery.The country\'s tax agency gave no date for the 25 percent increase...'
ANNOTATE =  sNLP.annotate(text)
POS = sNLP.pos(text)
TOKENS = sNLP.word_tokenize(text)
NER = sNLP.ner(text)
PARSE = sNLP.parse(text)
DEP_PARSE = sNLP.dependency_parse(text)

I am only interested in Entity Recognition which is being saved in the variable NER. The command NER is giving the following result

The same thing if I run on Stanford Website, the output for NER is

There are 2 problems with my Python Code:

1. '$' and '50-billion' should be combined and named a single entity.
Similarly, I want '25' and 'percent' as a single entity as it is showing in the online stanford output.
2.  In my output, 'Washington' is shown as State and 'China' is shown as Country. I want both of them to be shown as 'Loc' as in the stanford website output. The possible solution to this problem lies in the documentation .

But I don't know which model am I using and how to change the model.


texts: Handle 50,000 classes in OneVsRestClassifier

texts:
I'm new to data science and NLP. I'm trying to solve a problem that is having 1 million rows and some 50,000 distinct classes. The dataset has some text column as a predictor and the other one is the multilabel responses. I have been using tfidf to represent the text fields and MultiLabelBinarizer to transform the labels. But MultiLabelBinarizer is giving MemoryError.

And there is no way I can pass the legacy multi-label data representation using a sequence of sequences as it seems no more to be supported in the sklearn package. So, what should be my approach?

Any help is appreciated. Thanks in advance.


texts: What is the standard for dealing with null values in plotting?

texts:
There's really no "standard," I assume, but how do you deal with null values when plotting data? An example of what I'm talking about is listed below:

Suppose I'm plotting data on number of toys by color. Nearly half of these toys in my database have null values for color. Do you guys include the toys with null values in your visualization? How do you address that issue? Saying "nearly half of the toys didn't have colors listed" doesn't really add much to the conversation.


texts:
I have read lot of blogs\article on how different type of industries are using Big Data Analytic. But most of these article fails to mention

What kinda data these companies used. What was the size of the data
What kinda of tools technologies they used to process the data
What was the problem they were facing and how the insight they got the data helped them to resolve the issue.
How they selected the tool\technology to suit their need.
What kinda pattern they identified from the data & what kind of patterns they were looking from the data.

I wonder if someone can provide me answer to all these questions or a link which at-least answer some of the the questions. I am looking for real world example.

It would be great if someone share how finance industry is making use of Big Data Analytic.