texts: Number of parameters keras dense layer with a 2D input

texts:
I am using 2D data in a classification problem using keras.

So I am defining a keras model as following:

in_ = Input((5, 10))
out = Dense(100, activation='relu', name = 'dense_1')(in_)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

which returns a compiled model with the following parameters:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_9 (InputLayer)         (None, 5, 10)             0
_________________________________________________________________
dense_1 (Dense)              (None, 5, 100)            1100
=================================================================
Total params: 1,100
Trainable params: 1,100
Non-trainable params: 0
_________________________________________________________________

What I don't understand is why the dense_1 layer has only 1100 parameters and not 5100 parameters.
What I was expecting is that the Dense Layer is going to connect to all the inputs 50 (5*10=50 inputs) giving a number of parameters of 5100 (100*50+100=5100, weights + biases). So apparently the Dense Layer only connects to the last dimension of the input? What happens in the other dimension?

If I flatten the input layer I obtain my expected number of parameters:

in_ = Input((5,10))
x = Flatten()(in_)
out = Dense(100, activation='relu', name = 'dense_1')(x)

model = Model(in_, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_13 (InputLayer)        (None, 5, 10)             0
_________________________________________________________________
flatten_6 (Flatten)          (None, 50)                0
_________________________________________________________________
dense_1 (Dense)              (None, 100)               5100
=================================================================
Total params: 5,100
Trainable params: 5,100
Non-trainable params: 0
_________________________________________________________________

So what is going on with a Dense Layer when the previous layer has more than one dimension? What happens with the dimensions and the dot products and biases? Why does the number of parameters changes?


texts: How to detect the begin word and end word in a sentence with machine learning

texts:
I have some English text that has been tokenized. For example, the length of text token is about 20000 and each word (tokenize) has an index. Also, each index has a label, as the beginning word in a sentence labeled as 'b', as an end word (include symbol) in the sentence labeled as 'e', with other words labeled as 'o'. There is labeled training data and unlabeled test data.

My question: how do we use a machine learning method or deep learning model to solve the issue that predicts the label of words in the test data?
I mean how do we make the label and data for training just a word for training? I am confused.


texts: Logistic Regression : Solving the cross-entropy cost function analytically

texts:
Logistic regression cost function is cross-entropy. It is defined as below:

This is a convex function. To reach the minimum, scikit-learn provides multiple types of solvers such  as : ‘liblinear’ library, ‘newton-cg’, ‘sag’ and ‘lbfgs’.

Is it possible to analytically find the minimum? if yes, what can we say of the computing complexity?


texts: Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees

texts: I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used the scikit-learn built-in feature importance), but with Deep Neural Networks I get very different results (used permutation feature importance). I also ran PCA, and PCA gave me similar results to DNN! Is this normal? Could both of them be accurate still? I have quite a large amount of initial features.


texts: Anomaly detection using k-means clustering in Python

texts:
I'm working on an anomaly detection task in Python.
Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values.

In order to find anomalies, I'm using the k-means clustering algorithm. I've split data set into train and test, and the test part is split itself in days.
The training is done using the train part of the data set and the prediction is done day by day.
I'm going to do like this because this will be the usage in production.

To distinguish if a record is anomalous or not, I calculate the distance between each point and its nearest centroid.

_clusters = self.km.predict(day)
centroids = self.km.cluster_centers_

# calculate the distance between each record and each centroid.
# the result is a matrix which has as column the id of centroid and rows are records.
# so each value is the distance of between record and centroid
distance_matrix = spatial.distance_matrix(day, centroids)

# save in nearest_distances, for each record, distance between each point and its nearest centroid
nearest_distances = []
for distance_per_cluster in distance_matrix:
nearest_distances.append(min(distance_per_cluster))

nearest_distances = pd.Series(nearest_distances)

Then, using a threshold, I find anomalies

self.outliers_fraction = 0.01
number_of_outliers = int(self.outliers_fraction * len(nearest_distances))
threshold = nearest_distances.nlargest(number_of_outliers).min()

day_df['anomaly'] = (nearest_distances >= threshold).astype(int)

This code works, but I have a high number of false positive.
Data sets are not labeled, but analyzing results it's quite obvious.
This because the threshold is set using outliers_fraction that equals to 0.01, but it is completely arbitrary.

Since I cannot know in advance which is the "correct" threshold, I would like to ask you if there is a better way to find anomalies, in this contest, using k-means clustering algorithm.


texts:
In the past days, I read up on the theory behind attention, when to apply it and what types there are. I think I have a decent first understanding of the concept, but now I would like to apply some of the insights I got to my own project and I find myself stuck with the implementation of attention in TF. (Quick Link to TF Attention)

The attention layer requires me to provide at least the queries and values. Correct me if I am wrong already, but this is my idea of what they are:

Queries: These are the hidden states of my decoder
Values: These are the hidden states of my encoder

So far so good. The thing I am struggling with is the fact that I have no idea where the hidden states of my decoder might come from. I would like to implement a self-attention mechanism. So my decoder hidden states are generated dynamically and I cannot know them before actually applying the attention layer. The example provided in the docs was not helpful for me, because it focused on a problem where I already have some query sequence.

Apart from whether the mentioned TF attention layer is applicable for self-attention, how do I interpret the different inputs?


texts: Where can I find a social network image dataset?

texts:
I am supervising a programming project whose goal is to detect offensive images on social networks.
I would like to have a representative dataset of social network images. It would be best if the dataset were already classified. Otherwise, the classification of a (smallish) dataset could be a part of the project.

I looked around the internet and searched various dataset collections that are freely available, but I haven't been able to find anything appropriate.

What is the best way to get a large social network image dataset these days?


texts: How meaningful are the results when you difference the time series dataset before clustering?

texts:
On a certain task where I need to perform K-Means Time Series Clustering with DTW algorithm, I would like to know how credible the results are when performing clustering on the original vs a dataset where first and second order differencing has been applied.
The original dataset is highly non-stationary and high correlation is present between the variables, so I decided to make the dataset stationary by differencing and only then perform clustering. On the differenced dataset , the silhouette score for k=2  is 0.68 and for k=3 the score is 0.541, while on the original dataset for k=2 the score is 0.79, and for k=3 it  is 0.434. To summarize, I would like to know if:

Clustering results with non-stationary  time series variables, even if clustering evaluation score is higher less preferable than clustering results with stationary variables?
I also would like to know if applying differencing is a common methodology for such tasks, as I have not found enough relevant literature.

If you can, please feel free to provide links to relevant publications.


texts: how to take CSV file input in list of tuples

texts:
I have a .txt(data.txt) file containing csv data like:

X         Class
15.0001   Yes
18.00     NO
17.07     Yes

I need to make a function to return a list of tuples of each samples. So far I did:

import csv
def readAllData(str):
    with open(str,'r') as f:
        f.readline()
        data=[tuple(line) for line in csv.reader(f)]
    return (data)

Output:

[('15.001\tYES',),
 ('18.00\tNo',),
 ('17.07\tYes',),]

I want:

[(15.001, Yes), (18.00, No), (17.07, Yes)]


texts: RNN time-series predictions with multiple features containing non-numeric features and numeric features?

texts:
The question RNN's with multiple features is ambiguous and not explicitly in differentiating different features. I want to understand how to use RNN to predict time-series with multiple features containing non-numeric data as well. As a deep learning model, I assume I don't need to quantify the non-numeric elements.

Suppose sales data

Sales | Weather | Holiday | Temperature
100   | Windy   | Yes     | 3
2000  | Sunny   | Yes     | 20
200   | Sunny   | No      | 30
-5    | Stormy  | No      | 3
-50   | Cold    | No      | -50
500   | Cold    | Yes     | -20

where I want to predict the Sales column with the other columns. I have found demos such as here about using RNN with numeric data, without enriching the data with non-numeric data.

So

How can I predict time series with multiple features contain numeric features and non-numeric features?

Other interesting questions related to RNN

Multivariate time-series RNN -question in SO with numeric data
Time series prediction using ARIMA vs LSTM


texts:
I have a clarification. I have to create a classification model for certain set of documents. We are supposed to flag it anamoly or not based on certain terms in the document. My question is the terms maybe influenced by multiple parameters. Let us say the properties of the document will give different font sizes even though it refers to same term distributed throughout the document.
So let us assume I have a term1 occuring on the first page of the document, I will extract the properties of the font. Then again, I will look for same token and if the properties of the term1 does not match in the other pages, I will flag it as anamoly.
How do we represent them in form of feature vectors for classification model?
I need to incorporate the properties of the terms as one of the input vectors in the document.
Any help would be appreciated.


texts:
I'm wondering how Word2Vec is being constructed.

I've read tutorials simply stating that we can train a skip grams neural network model and use the weights that are trained as word vectors.

However, I've also seen this picture:

If reading this diagram correctly:

1) The CBOW and Skip Grams model are both trained with some inputs

2) The output of CBOW is used as an input to the middle neural network

3) The output of skip grams is used as an output to the middle neural network.

The output of CBOW is the a prediction of the center word given a context, and the output of skip grams is the prediction of the surrounding center word.

These outputs are then used to train another set of neural network.

Hence we first train the CBOW, and Skip-gram neural network, then train the middle neural network afterwards? And the input to the middle neural network is one hot encoded.

Is the above interpretation correct?


texts:
Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric approach.

On the other hand, I am currently dealing with a project that doesn't need any predictive models, but contains lots and lots of data to be explored, analyzed, summarized, and re-organized (rich geographic datasets can be quite multidimensional and messy). As the result the project directory looks approximately like this:

./DS-project
|
|__ data/
|
|__notebooks/
|    |
|    |__thing_to_analyze_1/
|    |    |
|    |    |__01_aa_analysis1.ipynb
|    |    |__01.1_aa_analysis1.ipynb
|    |    |__02_aa_analysis2.ipynb
|    |    ...
|    |
|    |__thing_to_analyze_2/
|    ...
|
|__src/
|    |
|    |__utility_package_1/
|    |    |
|    |    |__README.rst
|    |    |__setup.py
|    |    ...
|    |
|    |__utility_package_2/
|    ...
|
|__README.md
|__environement.yml

Currently the biggest pain is caused by the notebooks. We are following notebook naming convention suggested here, but due to the sheer number of thing_to_analyze_x folders and the number of notebooks in them, it can be quite hard to find previous research (even when notebook summaries are documented in some sort of README file).

Does anyone have experience dealing with such projects? Date based project/notebook naming seems quite interesting (e.g.: 2019-03-05_thing-to-analyze-1), but I'm afraid it might make it harder to understand which stuff is old and which stuff has been recently revisited.


texts: I've created masks(numpy array with 0,1 as values) and tried exporting this array to jpg using matplolib but it's not exporting the values as it is. I'm getting a range of pixel values in resulting image. Is there any way in which I can add the array values to pixel values as it is. For example numpy array is 40X40 and resulting jpg image resolution is 40X40 pixels.


texts: Value of AP and AR are -1.000 in evaluation

texts:
As I understand, the AP amd AR calculation are as of follow:
Precision = truepositive/ (truepositive + falsepositive)

Recall = truepositive/ (truepositive + falsenegative)

Technically, the lowest number they can be is 0, as in total fail in training and 1 as in perfect model. However, with my customer data set, my AP and AR have been consistently -1.000 when putting through different model for training. What can be the possible reason for this, wouldn't 0 the lowest they can go as I mentioned above?


texts: In a random forest, are all decision trees given same priority?

texts: In random forest algorithm, m (say) number of decision trees are generated using p ( $log_2 n$ +1, where n= number of features) randomly selected features. The label of any sample from test data is chosen by the number of votes by the result of these decision trees. But these decision trees result in different accuracy ( quite obvious). My question is, why should we give same priority to all the decision trees? If a decision tree has higher accuracy in predicting the labels of the test samples, shouldn't we give it more priority than those with lower prediction accuracy ?


texts:
I was training a binary classifier using XGBClassifier (basically boosted decision trees if I understand it correctly). I have 10K training examples. I have two distinct set of features (but they could be dependent), one contains 26 features (call the set A) and the other contains 96 features (call the set B). I tried to train 3 classifiers, each w/ a different combination of the feature sets, namely A, B and A+B. The result is that using only A is clearly better than using both A and B. At this point I thought it might be overfitting, so that using less number of features actually avoids the overfitting.

The # of trees used in the above trainings was 100. So I used 10-fold cross validation to find the optimal # of trees for each feature set combination, and they are all beyond 100 (like 300-500). So it seems to me the models were learning w/ less-than-desired degree of freedom and they were underfitting if you will.

So this confuses me: why providing additional features make things worse (i.e. using only A is better than using both A and B) when the models are on the under-fitting side (as opposed to overfitting)?

Or maybe a more general question: how do I find out the real problem in this situation?

Note: I have read this question about overfitting/underfitting and I still think my models should be underfitting, because apparently increasing model complexity helps (e.g. from 100 trees to 300-500 trees).


texts: Why doesn't LightGBM perform better than SVC (linear kernel)? [sentiment analysis]

texts:
I've read and heard about the mighty XGBoost which is one of the most famous models people are using today to solve Kaggle challenge. This makes me interest to develop my own intuition about the model so I decided to try XGBoost on my dataset, but unfortunately, I experienced a lot of dependency related problems when I tried to install XGBoost on mac :(

However, I did a bit more researches and found out another related library called LightGBM from Microsoft which they claim to achieve better result (or at least, equivalent) over XGBoost. Therefore, I decided to go with the LightGBM instead since I didn't encounter any problems during the library installation.

I expected that the model should outperform, or at least, perform similarly compared to other models on the same dataset. My dataset contains 14k of text documents and has 0 or 1 values which refer to positive and negative sentiment respectively as target variable.

As a result showing below, LightGBM with count vectorizer as input got the 6th place with an accuracy of 94.60%. I'm quite curious since the accuracy of LightGBM is not only lower than SVC but also lower than Extra Tree model (300 esitimators).

I don't have enough foundation and solid understanding behind these models so I couldn't come up with an answer about these outcomes. Can someone please give me an idea or assumption according to the result?

PS. All models was performed with 5-fold cross validation

model             score
--------------  -------
extraT           0.9528
svc              0.9514
sgd_elas         0.9481
extraT_tfidf     0.9476
svc_tfidf        0.9473
lightGBM         0.9460
sgd_elas_tfidf   0.9458
lightGBM_tfidf   0.9420
randomF          0.9409
randomF_tfidf    0.9345
mulNB            0.9307
berNB            0.9087
berNB_tfidf      0.9087
mulNB_tfidf      0.9036


texts: Running SQL-like queries over large schemaless JSON dataset in the cloud?

texts:
I've got about 5 million JSON files, about 50GB in total. They do not have a consistent schema (they're broadly the same format, but some have extra extension fields, some have missing fields, etc - the schema is quite complexly nested).

I would like to run SQL-like queries across these files - e.g. finding the count of files with a certain property, finding the count of files where property is in a numeric or time range, etc.

I have the files locally, and in an S3 bucket. I would ideally like to store the data in the cloud, so that others can run queries against them without having to have the files locally.

What are my options? Here are some solutions I've evaluated so far:

Apache Drill.

Pros: Great for SQL-like queries and 'on-discovery' schemas.
Cons: Ruled out because it struggles with a dataset this size both locally (runs out of memory) and hooked up to S3 (simply doesn't return results, I think because it has to get a local copy of the data).

Google BigQuery.

Pros: Great for SQL-like queries, stores in the cloud, large datasets, plus the friendly user interface is a bonus. JSON can be loaded from command line.
Cons: I'd need to define a schema, and remove or fill any data that doesn't conform to it. This is a hassle, and a problem because it means I can't query some data that I would like to (e.g. extension fields).

Amazon Athena.

Pros: Cloud support (runs against S3), and (I think) has SQL-like queries.
Cons: Seems to require a defined schema (one you define at runtime, rather than in advance)? Also seems to need the files in the query to match the schema, which wouldn't work for my data. However, I'm not sure this is correct, because the documentation is quite confusing.

Google Cloud Datastore.

Pros: SQL-like queries with GQL, stores in the cloud. NoSQL, so schemaless.
Cons: Seems to require pre-defined indexes in order to run queries (?), limited query support (e.g. no aggregation or joins). No documentation on how to load JSON files into it (possibly needs Cloud Dataflow setup).

I'm leaning towards BigQuery, mainly because it's much easier to get up and running than any other solution. However, the requirement for a predefined schema is a major hassle.

Are there any solutions I've missed? Have I misunderstood Athena's requirements about schemas?


texts: Differences of get_dummies and labelbinarizer?

texts: Are there any differences between get_dummies and labelbinarizer in terms of what they want to achieve? It seems to be both will somehow do a one-hot encoding.


texts: Problem with escape character in paste-function in R

texts:
I am trying to paste together a path-string. Like this "C:\testfile.txt"
This doesn't work at all:
filename <- "testfile.txt"

path <- paste("C:\\\",filename,sep="")

This also doesn't produce the right result:

filename <- "testfile.txt"

path <- paste("C:\\\\",filename,sep="")

[1] "C:\\\testfile.txt"

Can you help me to prudece a path-string with only one backslash?


texts: Many separation line using RBF kernel in SVM

texts:
Below is my code, it take a range of a number, creates a new column label  that contains either -1  or 1.

In case the number is higher than 14000 , we label it with -1 (outlier)
In case the number is lower than 14000 , we label it with 1 (normal)

## Here I just import all the libraries and import the column with my dataset
## Yes, I am trying to find anomalies using only the data from one column

df['label'] = [-1 if x >= 14000 else 1 for x in df['data_numbers']]  #What I explained above

data = df.drop('label',axis=1)
target = df['label']
outliers = df[df['label']==-1]

outliers = outliers.drop('label',axis=1)

from sklearn.model_selection import train_test_split
train_data, test_data, train_target, test_target = train_test_split(data, target, train_size = 0.8)
train_data.shape

nu = outliers.shape[0] / target.shape[0]
print("nu", nu)

model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00005)
model.fit(train_data)

from sklearn import metrics
preds = model.predict(train_data)
targs = train_target
print("accuracy: ", metrics.accuracy_score(targs, preds))
print("precision: ", metrics.precision_score(targs, preds))
print("recall: ", metrics.recall_score(targs, preds))
print("f1: ", metrics.f1_score(targs, preds))
print("area under curve (auc): ", metrics.roc_auc_score(targs, preds))
train_preds = preds

preds = model.predict(test_data)
targs = test_target
print("accuracy: ", metrics.accuracy_score(targs, preds))
print("precision: ", metrics.precision_score(targs, preds))
print("recall: ", metrics.recall_score(targs, preds))
print("f1: ", metrics.f1_score(targs, preds))
print("area under curve (auc): ", metrics.roc_auc_score(targs, preds))
test_preds = preds

from mlxtend.plotting import plot_decision_regions                                 # as rbf svm is used hence lot's of  decision boundaries are drawn unlike one in linear SVM
# the top one central points with blue quares are outlietrs while at the bottom they are orangy triangles(normal values)
plot_decision_regions(np.array(train_data), np.array(train_target), model)
plt.show()

Output from training data

accuracy:  0.9050484526414505
precision:  0.9974137931034482
recall:  0.907095256762054
f1:  0.9501129131595154
area under curve (auc):  0.5876939698444417

Output from test data

accuracy:  0.9043451078462019
precision:  1.0
recall:  0.9040752351097179
f1:  0.9496213368455713
area under curve (auc):  0.9520376175548589

My graph seems to be having so many sepearation lines, I was thinking I would only be getting one that differentiates between the outliers and the normal data.


texts: How to create a time series data with five minutes granularity

texts:
I have a dataset with the format

I need to create a new time series data based on the above dataset with 5 minutes granularity using python, the output should look like this where eventType will be binary(0/1) for a specific event (example- 8 only)

Can anybody help me to solve this?


texts: Is the GA R package the best Genetic Algorithm package?

texts:
I am learning about Data Science and I love the Healthcare part. That's why I have started a blog and my third entry is about using Genetic Algorithm for solving NP-Problems.
This post is https://datasciencecgp.wordpress.com/2015/01/31/the-amazing-genetic-algorithms/

I have some expertise with GA package solving problems like the TSP, but do you know any most powerful R package?

Thanks so much!


texts:
I implemented a fully connected MLP of shape [783 (input), 128 (hidden layer) and 10 (output)] the hidden layer had a sigmoid activation function and the output a sofmax.

I tested with the dataset of keras: Classify images of clothing.

At first I got the ouput was 0.1 at all the exits not matter the input. I then read this and because someone asked about the weights initialization I changed my weights initialization from a normal distribution between [0, 1) to [-1, 1). After that my network started working.

Why did this happen? I believe the prection of 0.1 is some kind of local minima because it just says the same probability for all, at least is what makes sense if you knew nothing about the data. But why? I would love to be refered to a paper that talks about this issue and how to prevent it because I am trying with another dataset now and I got the same problem (but this time I could not make it work. I even try Xavier initialization and still no good result).


texts: Do I use class weights to penalize false negatives or threshold optimization to improve recall?

texts:
I built a Random Forest model for a binary classification problem.Both the classes in the target variable are balanced. My main class of interest is 'class 1'.

False negatives are more costly to me, so it would make more sense to reduce false negatives by optimizing the recall.

I read online that I can assign weights to my classes to penalize false negatives ( basically misclassification of class1) or optimize the threshold using a precision-recall curve to improve the recall for class 1.

Im my case ( where both classes are almost balances) which method would be better?Can I still assign class weights to penalize false negatives?If yes, how do I determine what weights to apply to each class?or optimizing the threshold is better?

Here is my classification report:


texts:
All,
I am new to marketing data science. I have seen many literature on RFM modelling and do not understand why we put the aggregated scores/customer into quantiles using qcut.
e.g.
# --Calculate R and F groups--
# Create labels for Recency and Frequency
r_labels = range(4, 0, -1); f_labels = range(1, 5)
# Assign these labels to 4 equal percentile groups
r_groups = pd.qcut(data_process['Recency'], q=4, labels=r_labels)
# Assign these labels to 4 equal percentile groups
f_groups = pd.qcut(data_process['Frequency'], q=4, labels=f_labels)
# Create new columns R and F
data_process = data_process.assign(R = r_groups.values, F = f_groups.values)
data_process.head()

This is an example from : https://towardsdatascience.com/recency-frequency-monetary-model-with-python-and-how-sephora-uses-it-to-optimize-their-google-d6a0707c5f17 . What i don't understand is:

why do we use qcut? -- i know it means we get equal number of customers falling in each percentile/quantile but why would we want this. Is it because if our data is skewed we wouldn't want meaningless bins where there is 1/2 values in a bin, or is it to mask the affect of outliers?


texts: Idenitity between TD(0) algorithm and Policy Evaluation in Dynamic Programming when alpha is equal to 1

texts:
TD(0) algorithm is defined as the iterative update of the following:

$$ V(s) \leftarrow V(s) + \alpha({r + \gamma V(s')} - V(s) ) $$

Now, if we assume alpha to be equal to 1, we get the traditional Policy Evaluation formula in Dynamic programming. Is it correct?


texts:
I would like to analyze speech samples looking for speech-language pathologies.
Most of the resources I can find are about speech recognition which is a completely different problem.

I will always know what is the text that is being spoken on the given sample.
I can assume that audio samples will always be of good quality with no background noises.

I would like to construct a neural net(or maybe some different model) that will detect certain anomalies, but I have few questions.

How should I pass the input to the net? The input will have two parts: speech and text being spoken (+ label). Should I make I somehow divide these parts or just concatenate them as one sample?
What can I read to better understand working with audio/speech in ML environment?


texts: How to choose PCA or KernelPCA a priori?

texts:
I am learning about dimensionality reduction and I understood that one of the most used techniques in ML is PCA.

If I understood correctly, I use PCA whenever I want to reduce the number of features which should be mostly linearly separable (independent ?).

When the features are not-linearly separable (linearly not independent?), a nonlinear technique is required to reduce the dimensionality of a dataset and therefore I use KernelPCA.

Question: Supposing that what I just wrote is correct, how can I know in advance if the features are linearly or non-linearly separable before i decide which technique to use? So far the only way I was able to "guess" is by plotting the features after a PCA and checking if i can separate the new features through straight lines/surfaces. If I am not able to do so, then I apply Kernel PCA. Is this approach even correct?

Note:

Feel free to modify my question where the "?" are :)
My features don't have labels. It's an unsupervised problem.


texts: Code for Multivariate Random Forest in Python/R?

texts:
I am trying to code multivariate (or Multi output dx input features and dy outputs) Random Forest Regressor algorithm for a project, i.e. the algorithm can be used to predict more than one dependent variables using same features and training the model only once to account for correlation between the response variables. Does anyone know or have an already coded algorithm in R or python that I can use as a reference?

I know that Sklearn Randomforest can be applied in Multivariate settings. And there is a Package in R called Mutlivariate Random Forest for such use. But, I need an amateur level from scratch implementation that I can understand and learn from about how to code GINI gain function and prediction function for the algorithm.

I tried scouting the Github, but haven't found anything useful yet.

Thank you in advance to whoever is helping.


texts: kmeans cluster results for use as xgboost feature

texts:
I am curious if kmeans clusters can be used as xgboost features, along with the original features?
Specifically for features X and labels Y can we:

Split into X_train, y_train & X_test, y_test
Compute the optimal number of k for X_train, y_train
Add a new feature to X_train which is the insample cluster in 1,...,k
Train an xgboost on X_train + cluster, y_train
Test model performance on X_test + out of sample cluster, y_test

Is there any standard approach in the literature?


texts:
I'm trying to find all grammatical forms for words from a dataset. I'm new to NLP.

I'm using a combo: spaCy (to get a base form of a word), word_forms (to get all possible forms). Sadly, it seems that they are not able to get everything right. Examples like 'wildest' or 'scariest' seem to be out of reach.

Maybe I'm expecting too much (in most of the cases the solution works really well).

Is there a decent tool I could use? I'm aware of nltk lemmatizer, its performance is worse than word_forms and spaCy. I'd appreciate some kind of advice.


texts: Depending upon how I download, I get two different files

texts:
I am downloading the data set for the Kaggle competition on the titanic.
If I use the following code :
if (!file.exists("data")){
  dir.create("data")
}
fileUrl <- 'https://www.kaggle.com/c/titanic/download/train.csv'

download.file(fileUrl, destfile='./data/train.csv')

I get a 14kb file, however,
Paste this Url directly in your browser and you will download the correct file about 60kb.


texts: Collapse a list to most common spellings

texts:
I have a user-generated list of proper names.  For the sake of conversation, imagine it's pet names.

This list will have many variations of the same name: misspellings, alternative spellings, and even random use of punctuation or spaces.

eg

Fluffy
Fluffy the Dog
Flufy Dog
Fllufy

I can get a count of each spelling to figure out that, say, Fluffy is the most common at 2,000 occurrences while the others are all <100.

I tried to condense using the Levenshein distance package (python-Levenshtein) which had mixed success.  I would assign a name to a "more likely" name if it's Levenshtein distance was < 3 and the "more likely" name had a higher count.  When I manually search a few known common names, I can manually weed out the Levenshein winners from the losers, as well as include others Levenshein missed in my list (eg Fluffy the Dog and Fluffy have a huge distance of 8).

Are there better techniques I should be trying?


texts: Predicting or patron find of a binary variable over time

texts:
I'm new to ML and trying to find some practical use to it I encountered with the chance of saving the connections and disconnections (the binary variable) of a bunch of users like this:

"User A connected at 10:02:33

User A discconected at 10:05:02"

I was wondering how could I determine whenever the user is going to connect again, analyzing his pasts connections, disconnections and time online.

My concers are the following:

Would this be posible?
What's the best method for this?
How much samples per user would I need?
What would be the best way to structure the data, and besides the user identifier, action(connect/disconnect) and time the action take place what other info would be useful?

An just as an extra question: Would it be possible to add more data in real-time to improve the algorithm prediction? How?


texts: Predict time to reaction (chemical engineering)?

texts:
I need a hint on the problem below. This is related to predictive analysis and chemical engineering. I don't background in chemical engineering, and that's why I am looking for some hints. I want to know if there's a technical term for the variable/problem I am working on. This would help better fine my search.
Imagine I have a fluid streamline (a production line with chemicals). On the input of that line, we pour some chemicals and change some parameters (temperature, pressure, ...etc) and we want to predict the time necessary to see change at the end of that line.
We can only control the input parameters, and we want to know how long would it take to see changes at the end of the pipeline (that delay time is the target var).
I know some parts of this will depend on the velocity and length of the production line, but also there will some reactions happening on the road. Anything rings the bill here? I am trying to read some literature review about this, if you have some useful information and know some helpful keys words, please let me know.


texts:
I try do to a neutral network with keras using R.
For that I use the function layer_text_vectorization(max_tokens, output_sequence_length). However, when I execute it, I get the following error:
Fehler in if(tensorflow::tf_version() < "2.1" stop ("Text Vectorization requires TensorFlow version >=2.1", : Argument hat Länge 0
I tried to install tensorflow with install_tensorflow(version="2.1") and install_keras(version="2.1") but if I try is_available("keras") it says FALSE, so I think something is going wrong there..
Anyone who could help me please?


texts: Deep Q-Learning for physical quantity: q-values distribution not as expected

texts:
Setting

I am trying to learn a specific physical quantity (radiance) inside a 3D scene with Deep Q-Learning. Just to give a quick overview, my agent shoots rays inside the scene: the reward is the irradiance of the points hit. This means that the reward is given only when a light source is hit - only 1% of the times. This lead to a very sparse reward function.

My state is a tuple of the 3D spatial coordinates inside the scene, my actions are the possible discrete directions used by the agent to scatter rays. The q-values represent this physical quantity based on that specific action(/direction).

Problem

I expect the q-values to be higher in the first 10 actions, and then slightly decrease. This would reflect the physics of my system. When the training starts, this is actually the case:

After some episodes, the q-value of one action starts to spike, as seen in the figure below. This does not reflect the physics of the environment, for which the incoming radiance should be distributed over all the actions.

EDIT:
The trend lines for the q-values can be seen below. Each line is the q-value for a specific action. During the last iterations the q-values for two specific actions explode.

I know about target networks and experience replay (uniform and PER), but this did not solve my issue.

Are there any methods that I can try to deal with this problem? Is this a problem of vanishing/exploding gradient? Thanks


texts: Classification problem: estimate the number of people in a household through account transactions

texts:
I want to estimate how many people are in a household by looking at account transactions. It would be also interesting to understand if in the household there are children.

I thought that a possible way to proceed could be to compare the living expenses with that of the average individual according to the national statistics. Moreover I could also perform text analytics to retrieve informations about what products people bought (in order to understand if they spent money to buy children products).

As model I could maybe use a random forest for classification (but please correct me if I'm wrong).

Are there better way to solve this problem?


texts: The embeddings of GoogleNews-vectors-negative300.bin is stemmed or not?

texts: I will use the embedding GoogleNews-vectors-negative300.bin data to do some project. Just want to know whether it was stemmed of the words?


texts: RBF kernel can classify two classes as in figure?

texts: As you can see, I have some points (belonging to red and blue class), and I would to use an RBF kernel but I think that an RBF kernel can make points linearly separable only if they are located in perfect circular way. In this case I don't know how modify the kernel (or which parameters use) for respect the "oval" aspect that these data have.


texts: Use sentiment analysis on word level (aspect level)

texts:
I am trying to use Orange 3.20 Text Mining nodes to classify sentiments using a lexicon approach (either using the Method from Liu Hu or Vader).

After selecting the columns, I can see the documents already classified according to positive, negative, neutral or compound (in the example for Vader).

However, I was also trying to extract the most positive and negative n-terms from the documents. An example would be to classify for example the word "staff" that appears in multiple documents according to its average compound score. For example, "staff" could appear as "amazing" in one document (high compound score) and "awful" in another document, and the average in this case could almost be neutral. That would be very important to understand the specific positive/negative drivers in the text.

I'm planning to use Orange for teaching and I do this today with MeaningCloud or Lexalytics and I think it's the only thing that I still cannot do in Orange (without using Python). But probably I'm just new to Orange and not being able to understand how to do it.

Can I do this in Orange without resorting to Python?


texts:
I'm working on a problem where I have images(600) and corresponding scores. I have used VGG19 pretrained architecture where I connected the last CNN layer of VGG19 with 3 new fully dense layer. As a general process I'm doing the whole model training Vgg19(incremental) + 3 FC layers.
As a part of training exercise, initially I divided the scores in to few classes and trained the same architecture as a classficaion model. Accuracy is close to 99% on testing data as well.
Now if I'm using the same architecture for regression problem (after adjusting the necessary changes), I observed altogether different behavior.
From the first epoch itself, even for the training data it always predict the same outcome. It get changed with every epoch, but predict the the same logit for all the enteries in the dataset.
I'm not sure what could be the reason.
Please provide your answer.


texts: (CNN+)RNN-HMM hybrid for learning phonemes from a spectogram

texts:
I am currently working on a speech recognition task, on applying deep learning onto the standard acoustic model (gmm-hmm).

I've currently generated a spectrogram of my utterances, and using simple pattern recognition managed to receive a 40%WER on a yes/no dataset.. Not great but a start.  The CNN is being fed a context window of 40 frames, in which the center frame is being detected for, my question is whether the use of an RNN could benefit here? so that the RNN handles the context, and the CNN does "image analysis" on one frame spectrogram.

and if so will it implementation wise cause some problems, When I was doing it with a CNN, was the context dependency resolved by doing pattern recognition on a bigger part of the spectrogram, depending of the context window size, but introducing RNN, CNN only has to do analysis on one frame (which I see can become to even get a proper results from) at the time, and can the one framed information be piped to the RNN until a certain context size has reached? and if so how?


texts:
Need lesser memory consumption than doing sklearn.decomposition.TruncatedSVD on 99999 variables.

So I know that I can use TruncatedSVD to do the dimensionality reduction. However, my problem is about choosing the correct amount of n_components.

One approach suggested doing TruncatedSVD to the maximum amount of variables first and then calculate on this, how many variables on needs to reach a "explained variance" goal.

But doing the initial TSVD on 99999 variables seems to 1) take a long time (it hasn't finished it) and it 2) slowly seems to occupy all my memory (which I have 120GB).


texts: Confusion matrix - classes get confused with each other

texts:
I got an example from this book on page 318: Chemometrics for Pattern Recognition By Richard G. Brereton. The example explained that; classes A and B are more often mistaken for each other than they are for classes C and D,..., as they may be more closely related or even can be better viewed as one single group.

Given a confusion matrix and assign some values to each class. How do you know which classes get confused with each other the most? Also, what it means by closely related or eve can be better viewed as one single group? How are A and B closely related and viewed as one single group in this example? Based on the value in the confusion matrix?

Example:

Chemometrics for Pattern Recognition
By Richard G. Brereton - Google Book


texts: List of Correlations to Correlation Matrix

texts:
I am using python to do some data analysis and I need to represent the following table as a correlation matrix. The correlation value is a value between -1 and 1.

+-------------------+----------------------------------+--------------------------+
| Correlation Value |            Variable 1            |        Variable 2        |
+-------------------+----------------------------------+--------------------------+
| -0.07035          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | COVER_OPEN_WATER_PERCENT |
| nan               | ELECTRICAL_CONDUCTIVITY_AT_10_CM | Heat_1988-2013_trnd      |
| nan               | ELECTRICAL_CONDUCTIVITY_AT_10_CM | 1972                     |
| -0.059789         | ELECTRICAL_CONDUCTIVITY_AT_10_CM | BEO_HC14                 |
| nan               | ELECTRICAL_CONDUCTIVITY_AT_10_CM | st_6_avg                 |
| 0.214723          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | slope_angle              |
| 0.247674          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | total_mn                 |
| 0.01978           | ELECTRICAL_CONDUCTIVITY_AT_10_CM | FD13_low_flux            |
| 0.136542          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | 19.1.1                   |
+-------------------+----------------------------------+--------------------------+

What tool would be best to accomplish this task? I attempted to use matplotlib to create a 2d histogram but as far as I could tell it requires 2 numerical numpy arrays. Are there any other tools or functions that could create a 2d histogram depending on the level of the correlation value?


texts: Quadratic programming and Lagrange multiplier in SVM

texts:
I am a little confused because for some simple functions and constraints, using the Lagrange multiplier will be able to solve for the variables. However, in the SVM Lagrange expression, I learned that we had to use quadratic programming to solve for the maximum AFTER we express it in terms of Lagrange expression with some constraints:

Does that mean this problem can't be solved only using the Lagrange method? I'm not familiar with quadratic programming or the Lagrange multiplier. Could someone please kindly explain the relationship between the two? Preferably in the context of Support Vector Machines. Thank you so much!


texts: Is supervised machine learning by definition predictive?

texts: I am trying to organize a cheat sheet of sorts for data science, and I am working with the basic distinction between description, inference, and prediction. As examples of the first I see unsupervised methods described, and for the last I see supervised methods. So my question is simply, do these two sets of categories align? Is unsupervised to supervised as description is to prediction?


texts: ValueError: could not convert string to float: 'Nor967.jpg'

texts:
Whenever I try to use the data augmentation ImageDataGenerator I'm getting an error like could not convert string to float. Here is my code.

import numpy as np
import os
import cv2
import matplotlib.pyplot as plt

from pathlib import Path

jpeg_images =
list(Path(r'D:\covid_deep_learning\train\train').glob('**/*.jpg'))
np.array([np.array(cv2.imread(str(file))).flatten() for file in jpeg_images])
folder = ['COVID19_AND_PNEUMONIA','NORMAL']
Path = r'D:\covid_deep_learning\train\train'
for i in range(2):
    listing = os.listdir(Path+'/'+folder[i])
    for file in listing :
        img = cv2.imread(Path+'/'+folder[i]+'/'+file)
        resize=cv2.resize(img,(70,70))
        cv2.imwrite(Path+'/'+folder[i]+'/'+file,resize)
        cv2.imshow('resize', img)
        plt.imshow(resize)
    print(listing)
    print(len(listing))
def load_images(path, df):
    train_image = []
    for i in tqdm(range(df.shape[0])):
        try:

            train_image.append(resize)
        except OSError:
            print(df['id'][i])
    image_array = np.array(train_image)
    return image_array

train_image_path = r'D:\covid_deep_learning\train\train/'
test_image_path  = r'D:\covid_deep_learning\test/'

X = load_images(train_image_path,train)
test_images = load_images(test_image_path, test)

 X_train, X_test, y_train, y_test_class = train_test_split(X, y, random_state=42, test_size=0.2)
 y_train = pd.get_dummies(y_train)
 y_test = pd.get_dummies(y_test_class)
 model = Sequential()

model.add(Conv2D(32, (3, 3), padding="same",  activation='relu',input_shape=(70,70,3)))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))

model.add(Conv2D(64, (3, 3), padding="same", activation='relu' ))
model.add(BatchNormalization())
model.add(Conv2D(64, (3, 3), padding="same", activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Conv2D(128, (3, 3), padding="same", activation='relu' ))
model.add(BatchNormalization())
model.add(Conv2D(128, (3, 3), padding="same", activation='relu' ))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())

model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(256, activation='relu'))
model.add(Dropout(0.50))

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.25))

model.add(Dense(2, activation='softmax'))
model.summary()
opt = SGD(lr=1e-3, momentum=0.9, decay=1e-3 / 25)
model.compile(loss='binary_crossentropy',optimizer=opt,metrics=['accuracy'])
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img

datagen = ImageDataGenerator(
    rotation_range=10,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest')

datagen.fit(X_train)
print('augmentation')

After data augmentation I'm getting an error like this:

ValueError                                Traceback (most recent call last)
<ipython-input-61-f6bc9e2818e8> in <module>
     11         fill_mode='nearest')
     12
 ---> 13 datagen.fit(X_train)
     14 print('augmentation')

~\Anaconda3\lib\site-packages\keras_preprocessing\image\image_data_generator.py in fit(self, x, augment, rounds, seed)
    924             seed: Int (default: None). Random seed.
    925        """
--> 926         x = np.asarray(x, dtype=self.dtype)
    927         if x.ndim != 4:
    928             raise ValueError('Input to `.fit()` should have rank 4. '

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
    536
    537     """
--> 538     return array(a, dtype, copy=False, order=order)
    539
    540

ValueError: could not convert string to float: 'NORMAL2-IM-0340-0001.jpeg'


texts: Can cosine similarity be applied to multidimensional matrices?

texts:
I'm trying to find the similarity between two 4D matrices. Because cosine similarity takes the dot product of the input matrices, the result is inevitably a matrix. Is there a way to get a scalar value instead? Could inner product used instead of dot product? That is, is

cossim(A,B) = inner(A,B) / (norm(A) * norm(B))

valid? Or is there a better way to find the similarity between multidimensional matrices?


texts:
I was hoping to get some help on a task I have been given. I have a data set with the chemical breakdown of a drink. My job is to find some sort of analysis with the given data. The data has 10 different attributes and a numerical value associated with each attribute and roughly 300 instances of it. I have to use Weka and perform some sort of Clustering, Association, or Classification algorithm. I cannot think of a good analysis that can be done with this data!

At first I thought of using the clustering algorithm but that does not tell me much besides clustering the the data into the number of clusters I assign and telling me the mean composition of each chemical per cluster. Not much of analysis there.

I was hoping someone might be able to guide me in coming up with a good analysis prompt I can do with this data!


texts:
So, i was learning the KNN Algorithm and there i learnt cross Validation to find a optimal value of k.Now i want to apply grid search to get the optimal value.I found an answer on stack overflow where both standardScaler and KNN are passed as estimator.

 pipe = Pipeline([
        ('sc', StandardScaler()),
        ('knn', KNeighborsClassifier(algorithm='brute'))
    ])
    params = {
        'knn__n_neighbors': [3, 5, 7, 9, 11] # usually odd numbers
    }
    clf = GridSearchCV(estimator=pipe,
                      param_grid=params,
                      cv=5,
                      return_train_score=True) # Turn on cv train scores
    clf.fit(X, y)

My questions

I am already applying the standardscaler to standardize the data before passing to KNN. So here do i still need to pass the standardscaler in the estimator?
why X and Y are passed instead of x_train and y_train assuming x and y are independent and dependent variable and x_train,y_train are formed after train_test_split operation ?

Any example of such code will be appericiated.


texts: Counting the number of layers in a neural network

texts:
I am going over the Udacity tutorial on Neural Networks.

Here's a diagram from the tutorial:

What makes this a '2 layer neural network'?

I was under the impression that the first layer, the actual input, should be considered a layer and included in the count.
This screenshot shows 2 matrix multiplies and 1 layer of ReLu's. To me this looks like 3 layers. There are arrows pointing from one to another, indicating they are separate. Include the input layer, and this looks like a 4 layer NN.


texts: How to prepare photo data for training model to recognize bowling ball name, brand and manufacturer from photo of bowling ball?

texts:
I am asked to do this. The client can only prepare a only one photo for each ball from the product page of bowling manufacturer.
However, I need huge amount of image data for each ball.

Here is what I am thinking. To fake thousands pieces of photo data, turn the ball into 3D model and take thousands of photo of that ball from different perspective in software like Maya or 3DS Max.

I am not sure if this would work for detecting balls in reality since the source of data is not real.

Would my approach of preparation of image data from virtual environment work for training model for detecting objects in reality?

BTW, I do not have specialty in data science. I just have very fundamental concepts of ML. What algorithm would you suggest for my case?

Thanks!


texts: suggest a reinforcement learning agent that will learn to efficiently switch on a bit vector

texts:
The following env describes a simple task: there's a bit-vector of size 10, with all bits initially set to 0.

The agent must learn to switch it all to 1s. It has 10 possible actions - each one switches the corresponding bit to 1. Switching a bit which was 0 to 1 gives a reward of 1, but trying to switch a bit which is already 1 gives a reward of -1 and ends the game. The agent sees the entire bit-vector after each step.

Why isn't it very very simple? I wrote a policy-gradient approach agent but it doesn't arrive to sufficient results (stay at around ~3 points average) after 10K episodes.

What am I missing? Is it indeed such a hard task???


texts: How to use Chi-square test in dataset with negative values

texts:
I could not fully explain the title. In order to use the Chi-square test in my dataset, I am finding the smallest value and add each cell with that value. (for example, the range of data here is [-8,11] so I added +8 to each cell and the range turned to [0,19]).

dataValues variable is DataFrame type that holds my all data and contains ~2000 features, ~1000 rows, dataTargetEncoded variable is Array type that contains results as 0 and 1.

for i in range(len(dataValues.index)):
    for j in range(len(dataValues.columns)):
        dataValues.iat[i, j] += 8

#fit chi2 and get results
bestfeatures = SelectKBest(score_func=chi2, k="all")
fit = bestfeatures.fit(dataValues, dataTargetEncoded)
feat_importances = pd.Series(fit.scores_, index=dataValues.columns)

#print top 10 feature
print(feat_importances.nlargest(10).index.values)

# back to normal
for i in range(len(dataValues.index)):
    for j in range(len(dataValues.columns)):
        dataValues.iat[i, j] -= 8

But this causes performance problems. Another solution I'm thinking of is to normalize it. I wrote a function that looks like this:

def normalization(df):
    from sklearn import preprocessing

    x = df.values  # returns a numpy array
    min_max_scaler = preprocessing.MinMaxScaler()
    x_scaled = min_max_scaler.fit_transform(x)
    df = pd.DataFrame(x_scaled, columns=df.columns)

return df

My program has accelerated lots, but this time my accuracy has decreased. The feature selection process I have done with the first method produces 0.85 accuracy results, this time I am producing 0.70 accuracy.

I want to get rid of this primitive method, but I also want accuracy to remain constant. How do I proceed? Thank you in advance


texts: What's the difference between keras api augmentation and data augmentation definition?

texts: The augmentation definition is increasing the number of images by using rotation, crop and flip to avoid overfitting. The keras API apply augmentation but no increasing the number of image. What keras augmentation does in images? Is API augmentation such as preprocessing of images? Is augmentation replace the original image with new augmented images?


texts: Interpreting the evaluation result of multiple linear regression

texts:
I am learning the multiple linear regression model.
I've built a model and using R command:

summary(model)

I got this result:

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 253.2 on 44 degrees of freedom Multiple
R-squared:  0.3336, Adjusted R-squared:  0.2579  F-statistic: 4.405 on
5 and 44 DF,  p-value: 0.002444

How can I interpret this result in order to have a decision regarding the goodness of the model? In specific, what is the 44 degrees of freedom means for this case?

Also, why do we have adjusted and multiple r squared parameters?


texts:
How is AdaBoost different from a Gradient Boosting algorithm since both of them use a Boosting technique?
I could not figure out actual difference between these both algorithms from a theory point of view.


texts: In Mini Batch Gradient Descent what happens to remaining examples

texts:
Suppose my dataset has 1000 samples (X=1000) . I choose batch size of 32.
As 1000 is not perfectly divisible by 32 , remainder is 8.
My question is what happens to the last 8 examples. Are they considered? If they are, will they effect the efficiency of my model.
def next_batch(X, y, batchSize):
     for i in np.arange(0, X.shape[0], batchSize):
           yield (X[i:i + batchSize], y[i:i + batchSize])

This code is from a book and according to me this code is not considering the last remaining data points


texts:
I am trying to predict/forecast salesperson performance weekly, monthly, quarterly, and yearly based on the products that they sold over 3 years. As part of this effort, I grouped their number of units sold by the state, location, stores, product types, and sale dates.

By using this information I grouped their id and month of sales and number of units sold. Example data is shown below.

For ARIMA modeling, does this format work well enough to predict/forecast salesperson performance weekly, monthly, quartely, and yearly?

Id      Month   Units
65381   201703  467.0
65381   201710  3.0
65381   201712  6.0
65381   201803  20.0
65381   201805  2.0
65381   201807  20.0
65381   201812  16.0
65381   201904  2.0


texts:
I am trying to convert an array containing images into a tensor, but I'm getting error "TypeError: Expected binary or unicode string, got array"

images.shape = (10,)

images[0].shape = (375,500,3) and
images[1].shape = (400,500,3).. i.e. the images[i] have various shapes.

imgs = tf.convert_to_tensor(images)

I'm not sure how to solve this, please help.


texts: Are there any APIs for financial pattern recognition?

texts:
I have seen this question on the site, however it is asking for the techniques that can be used for implementation. The thing I want to know is that, on the other hand, if are there any APIs or services for financial pattern recognition, i.e. capable of finding particular patterns that occur in a given set of data?

Think like that: I supply the data and if there are head-and-shoulders patterns detected in it; the API returns the index of the left/right shoulder etc.


texts: What is a better solution for text classification than use of perplexity

texts:
To classify some texts, I train a language model over a training set and then select the model which has the lowest perplexity on a given test sample as the class of that sample.
I would like to know what other methods or measures except perplexity can be used.


texts: Implementing DQN - Pylessons.com article - Queries

texts:
I have followed the below link to implement DQN Algorithm
https://pylessons.com/CartPole-reinforcement-learning/
Can someone explain me:

Why do we need to have else condition in act function during training? (Actually in my training, episode is ending successfully if I have only random action but not both random and predicted actions)

Why are we not just doing self.model.predict(state) in test function and instead looping several times again as in training?


texts:
Hi I'm currently trying to predict if an item will be successful in my store, this means (How much is going to sale in USD)
My training dataset contains many features:

Item name
Item weight
Item category
Item country of origin
Item sales overall
Item sales per store
Item rating
Item price

Etc....
Since I will be introducing a new item for sale, I know very little about this new item:

Item name
Item weight
Item category
Item country of origin
Item price

Not all the features present in training data/test data will be present when I will be making predictions. Is this normal in ML ? What is the rule of thumb when doing feature engineering for this type of cases.


texts: How to predict the temperature for new points?

texts:
Assume that we are going to predict the temperature change of a specific point in a liquid over time. We have the following as our observation:
We have the information about 200 different points in this liquid. Each point has a coordinate(for example $x,y,z$) and we know the temperature change of this point from $t = 0s$ to $t = 100s$. Given a new point $P = (x_0, y_0, z_0)$. How can we predict its temperature change over this period of time?( I mean from $t = 0s$ to $t = 100s$)
Assume that all the points in the liquid have the same starting temperature at $t = 0$
I thought maybe we can use RNNs to do this, but I am not sure how. Does any open-source implementation of this exist?


texts: Error encoding categorical features using sklearn pipelines

texts:
I am new to sklearn pipelines and am using this post as a guide for my code:

https://www.codementor.io/bruce3557/beautiful-machine-learning-pipeline-with-scikit-learn-uiqapbxuj

I am trying to encode a categorical features using a transformation pipeline, but no matter what encoder I use, I get the same error. As far as I can tell from reading other posts, scikit-learn should be able to handle categorical variables as strings from version 0.20 or greater (namely with the OneHotEncoder.)

ValueError: could not convert string to float: 'Male'

Where I have entered xxxxxxxxxxx below replace with one of the following

ce.OneHotEncoder
ce.TargetEncoder
OneHotEncoder
OrdinalEncoder

from sklearn.pipeline import FeatureUnion
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import Imputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer
import category_encoders as ce
from sklearn.preprocessing import OrdinalEncoder
from sklearn.pipeline import Pipeline
import pandas as pd
import numpy as np

# create example data
example_df = pd.DataFrame({'Sex':['Male','Female','Female'],'Survived':[1,1,0]})
X_train = example_df.drop('Survived', axis=1)
y_train = pd.DataFrame(example_df['Survived'])

# build example pipeline
cat_pipe = ("categorical_features", ColumnTransformer([
            ("categorical", Pipeline(steps=[
                ("impute_stage", Imputer(missing_values=np.nan, strategy="median")),
                ("label_encoder", xxxxxxxxxxx())]), ["Sex"]
            )])
          )

example_pipeline = Pipeline(steps=[cat_pipe])

# fit pipeline
example_pipeline.fit(X_train, y_train)

# Name                    Version
scikit-learn              0.20.3
category_encoders         1.3.0
numpy                     1.16.2
pandas                    0.24.2


texts:
I have a dataset with age + another 14 variables. I have created 13 bins representing different age groups like so:
data["age_bins"] = pd.cut(data["age"], [16,20,25,30,35,40,45,50,55,60,65,70,75,80])

Then calculated the mean value of those 14 other variables per age group like so
data_age_bins_means = data.groupby(["age_bins"]).mean()

resulting in a 14 by 13 DataFrame called data_age_bins_means
Finally, I want to output a data structure with the 5 variables with the greatest mean value in descending order per each age group i.e. first sort each age group column separately and then choose those five variables with the greatest mean values for each age group. I was thinking about a MultiIndex solution but would badly need some help on a neat solution here. Many thanks!
P.s. I finally want to save that data structure to .json for easy loading to JavaScript


texts: sklearn.accuracy_score(y_test, y_predict) vs np.mean(y_predict == y_test)

texts:
What is the difference between these two methods for finding model accuracy?

I have used both methods in python3 and i normally get identical results. However in few cases i get completely different results, so I am trying to figure out the possible reason for this.


texts: Translating a Business Metric into an ML Accuracy Metric

texts:
Let's say that I'm building a binary classifier for giving loans to consumers. But apart from the standard accuracy metrics I can use, I have a requirement from the business, which wants to be able to "auto-approve" 80% of the loans.

My gut instinct is that this should be 1 - False Positive Rate, but I'm not quite sure.

Any ideas?


texts: Is there a Python memoization library that supports version control?

texts:
I'm doing a data science project in Python. I need to cache results to reduce computation time.

In my code I'll use function foo to calculate y = foo(x) for different sets of parameters x. In addition to different parameters x, I an often modifying foo too. For example, I'll have one version of foo in git branch branch-1 and another version of foo in git branch branch-2. I'll jump back-and-forth between these branches many times in a workday. I need a memoization library that caches the results of branch-1 and branch-2 separately iff foo has been called with different parameters or foo's source code differs between branch-1 and branch-2.

In other words, I need a library the following features.

Computations are saved to the cache.
Computations are not recomputed if they are in the cache.
Computations are recomputed if they are not in the cache or the code generating them has been updated.
Cache results are written to disk so they survive between different runs of my program.
Cache results for outdated code are preserved in case I revert the code with git.

A few important points

I'm okay with a system that detects whether the file containing soo has changed in place of detecting whether soo's function definition has changed. But in that case "changed" has to depend solely on git commit history. It can't depend merely on unix timestamps because I use git and git does not preserve unix timestamps.
I'm okay not using git history for versioning. It just seemed the most obvious approach to me.

Does a library exist that to do this?

I've already examined the following memoization libraries.

functools.lru_cache and python-memoization don't work because neither of them write results to disk.
Redis seems designed for web apps. Therefore I expect Redis is not designed to preserve caches for anything but the newest code.


texts: Differentiating roadmap of a loss function

texts:
Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$.

Binary cross entropy error:

$$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*ln(1-p(x_{i}))$$

The derivative of such roadmap would be:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}$$

We can use a chain rule to solve this:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}=\frac{\partial L(y|p(x_{i}))}{p_{i}}*\frac{\partial p_i}{w_1}$$

First term is easily understandable, it's just a differentiation of a loss function with respect to the estimated probability:

$$\frac{y_i -p_i}{p_i(1 - p_i)}$$

But the second term seems a little weird, from my understanding $p_i$ is just estimation of Bernoulli distribution (therefore it should be Empirical distribution). Somehow it's derivative is:

$$p_{i}*(1-p_i)*x_i$$

This is definitely somehow associated with the Bernoulli distribution, but how exactly? What is the definition of $p_i$ in this particular case?

Thank you!


texts: Connecting Authors with Published Papers

texts:
I'm specifically interested in tying doctors to their published papers. The key issue is that using name alone will result many collisions. I'm wondering what set of features I would need to reliably connect a doctor with a given published paper? Aside from weak features such as specialty, is there any database that has a link between doctor NPI and papers published?

I've seen this on linking NPIs to authors in PubMed but it seems rather unreliable.


texts: How to optimize XGBoost performance accuracy?

texts: I have dataset to predict customers dropout(yes,no), with 5 numerical features and 2 categorical features. I have applied a scaler to the numerical data and transformed the categorical features into dummies variables, creating 29 features. My dataset has shape of 6552 rows and 34 features. What is the recommend approach to tune the parameters of XGBClassifier, since I created the model using default values, i.e., model=XGBClassifier()? Should I use a brute-force looping the values in some parameters until I find a optimal prediction value? In this case what is recommended?


texts: Deep Reinforcement Learning with Space Invaders

texts: I want to better understand Deep Reinforcement Learning so I developed the Space Invaders game from scratch with Pygame. I have a fixed number of enemies (10). Instead of defining the states as a stack of frames (pixels) as in the state of the art, I rather defined them as relative distance between the player and the enemies + relative distance between the player and 5 closest enemy lasers. So the observation space dimension is (10+5)*2 (2 for x and y coordinates). I'm tuning the model for the moment and since I don't see any progression in the behavior (total reward is not really progressing after about 700 episodes), I thought that, beside other hyper parameters, maybe something is wrong with the way I defined the states and I want to have your opinion about it?


texts: How does inception decrease the computational cost?

texts:
From the second paragraph of 3.1 Factorization into smaller convolution in the paper Rethinking the inception architecture for computer vision:

  This setup clearly reduces the parameter count by shar- ing the weights between adjacent tiles. To analyze the expected computational cost savings,

  ...

  This way, we end up with a net 9+9 × reduction of computation, resulting
  25 in a relative gain of 28% by this factorization

Apprantly this design decreases the number of parameters. But I can't understand why it decreases the computational cost?

For the case of using 2 3*3 convnets to replace 5*5, I think it increases (3*3*9+3*3)/5*5 = 3.6 times of computational cost.

What do I miss here?


texts: Which Technique should we use for predicting an integer output?

texts:
I'm working on a problem where my target feature of type integer. i.e (n_clicks). In general, if we want to predict categorical target feature then we use classification algorithms and on the other hand for predicting a target feature of type continuous then we use regression techniques where my output will be -infinity to +infinity. But in my case if I use regression then my output will become a float value. One solution here would be convert the output float values into into integer or by use of pandas.round() to round figure the value.

Is there any alternative way to predict my output variable as an integer ?

Thanks

UPDATED: As suggested to use POISSON Regression, tried below code but still the output is same. Below is code snippet:

import statsmodels.api as sm
x_train_sm = sm.add_constant(x_train)
poisson_model = sm.GLM(y_train, x_train_sm, family=sm.families.Poisson()).fit()


texts: Error "comparison (6) is possible only for atomic and list types" encountered in metafor

texts:
I'm very new to r and trying to run a multi-level meta analysis using pre-calculated effect sizes. The data file can be accessed via this link:testrunfile
The script I used as a first step to fit the model was:
res <- rma.mv (yi = es_r, v = var, data = testrun, method = "REML", level = 95, digits = 7, slab = ref, random = ~ 1 | samp_id)

But I keep getting this error:
Error in verbose > 2 :  comparison (6) is possible only for atomic and list types
Could someone help give me a hand to help me understand what this error message means and where did I go wrong with the script?
Many thanks!


texts: Adding more features in SVC leading to worse performance, even w/ regularization

texts:
I have a relatively small dataset of 30 samples with binary labels (16 positive and 14 negative).  I also have five continuous features for each these samples.  I'm trying to use the support-vector classifier (SVC) for this task.  I tested the performance of different feature combinations and regularization strengths in the classification task using leave-one-out cross-validation.

One odd thing that I found is that if I took feature A and used it alone for classification, I might get, say 87% classification accuracy.  If I use feature B in isolation, I might get 60% classification accuracy (i.e., same as majority classifier baseline).  But then combining all the features, I would get only 63% classification accuracy.  This is despite performing a search across a large range of regularization strengths.

In case it matters, I'm using the sklearn SVC implementation, and varying the regularization parameter C.

Is this sort of behavior typical with an SVC classifier?  I'm not too familiar with this support vector algorithms in general.


texts: What should I care about while stacking as an ensemble method?

texts:
I'm using SMO, Logistic Regression, Bayesian Network and Simple CART algorithms for classification. Results form WEKA:

Algorithm               Sensitivity (%)       Specificity (%)         Overall accuracy (%)
Bayesian Network            57.49                 76.09                    65.24
Logistic Regression         64.73                 69.86                    66.87
SMO                         54.32                 79.20                    64.69
Simple CART                 71.88                 61.51                    67.56

SMO gives the best result for my classification problem, since it correctly classify the 79.20% of the class which is important for me. I want to increase this accuracy by stacking. I tried to combine some of them. In most of the cases I couldn't increase the accuracy but stacking SMO with Logistic Regression made a little increment in accuracy.

How can I explain why stacking SMO with Logistic Regression is better than others?

Is there any generalization such as combining tree classifiers gives good result in stacking? What should I care about while stacking?

EDIT:

                                Bayesian Network    Logistic Reg.   SMO         CART
Kappa statistic                   0.3196             0.3367         0.3158      0.3335
Mean absolute error               0.3517             0.4164         0.3531      0.4107
Root mean squared error           0.5488             0.4548         0.5942      0.4547
Relative absolute error (%)      72.3389              85.65        72.6299      84.477
Root relative squared error (%) 111.3076            92.2452       120.5239     92.2318
Weighted Avg. of F-Measure        0.653               0.671          0.676     92.2318
ROC Area                          0.725               0.727          0.668       0.721

Total number of instance is 25106. 14641 of them is class a, and 10465 of them belong to class b.

=== Confusion Matrix of Simple CART ===
     a     b   <-- classified as
 10524  4117 |     a = 0
  4028  6437 |     b = 1

=== Confusion Matrix of SMO ===

    a    b   <-- classified as
 7953 6688 |    a = 0
 2177 8288 |    b = 1

=== Confusion Matrix of Logistic Regression ===

    a    b   <-- classified as
 9477 5164 |    a = 0
 3154 7311 |    b = 1

Since SMO is successful at class b and CART is successful at class a, I tried to ensemble these two algorithms. But I couldn't increase the accuracy. Then I tried to combine SMO with Logistic Regression, the accuracy is increased a little bit. Why ensembling SMO with Logistic Regression is better than ensebling SMO with CART, is there any explanation?


texts: How to overcome the class imbalance issue in a text classification problem?

texts:
The dataset has 5 labels A,B,C,D,E

A,B,C are majority and D&E are minority.The penalty to misclassify D&E are huge. How can I implement a cost sensitive learning. The input to the model will be english sentences like user comments.


texts: Detecting address labels using Tensorflow Object Detection API

texts:
I am experimenting with the Tensorflow Object Detection API on a Windows 7 machine.  I am trying to detect US address labels (and similar blocks of text) as they appear on a piece of mail or an envelope. I am not trying to detect individual words or lines, but rather the full rectangular block of text. My address labels are typically isolated on the letter or envelope and they are surrounded by whitespace. For example:

I followed the tutorial to train a customer object detector. I used the pre-trained SSD Inception V2 COCO model along with 50 images of letters/envelopes containing address labels that were annotated with LabelImg. To annotate my objects (address labels), I drew the bounding box around the entire label with about 5-15 pixels of padding. After 200 training steps, I reached a loss of 2.5 and stopped the training.

Using the same tutorial, I obtained the trained inference graph. Finally, I adapted this tutorial for doing inference. I tested two images, once containing a dog and the other containing an address label. The dog was detected, but nothing was detected in my image with the address label was detected.

My questions are:

Is it reasonable to expect to detect the full block of text given it is surrounded by whitespace and not solid edges?
Am I annotating properly? I left about 5-15 pixels of padding between the address text and the bounding box when I annotated with LabelImg.
Do I have enough images for one-class detection in which the address labels vary little from letter to letter
Should I let the training go longer?


texts: Feature selection vs Feature extraction. Which to use when?

texts:
Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am  right.

Which technique would be preferred over the other and when?

I was thinking, since feature selection does not modify the original data and it's properties, I assume that you will use feature selection when it's important that the features you're training on be unchanged. But I can't imagine why you would want something like this..


texts: Orange Web Services - Problems installing imblearn module for Python Script

texts:
I am trying to implement SMOTE in Orange Web Services, but having troubles with the imblearn module.

Following screenshot is from Orange Python Script widget:

from imblearn import under_sampling, over_sampling
from imblearn.over_sampling import SMOTE

I can import it without problems with Python:


texts: Training my neural network to overfit my training dataset

texts: I want to train my neural net to overfit the training data. Should I just keep fitting my model with the same training data by using K-fold validation and setting epochs to infinity? Then, after it reaches nearly 100 percent prediction, I can just stop the training. Is it the right way to do it?


texts: I know how to split the dataset into train and test sets using train_test_split but is there any way that I can split the dataset into three different sets, i.e., "Train set", "Test set" and "Validation Set". An example should be enough.


texts: Type of images used to train a neural network

texts:
I am a newbie in neural network.

Saw this article Object detection with deep learning and OpenCV.
These three type of neural network are shortlisted in the article

Faster R-CNNs
You Only Look Once (YOLO)
Single Shot Detectors (SSDs)

Found a lot of online resources that help in understanding how the neural network actual works.

As building a neural network from scratch is time consuming and not entirely foolproof to get the desired efficiency.

I came across these articles Transfer learning & The art of using Pre-trained Models in Deep Learning,Transfer Learning and Transfer Learning, where we transfer the learning from an existing pre trained network to train the object which we would desire the network to detect.

All of these pre trained network have been trained on a data from either COCO or ImageNet or PASCAL VOC which contain different categories images.

Example case:

I want to train one of these to count the number of bananas here in this image

How should my training set of banana images be?

I need a fixed resolution of the image I feed into the network, so can I even feed a half banana like this for the training? I don't have to remove the background for training. Please correct me if I'm wrong.


texts: np.loadtxt function showing error, could not convert string to float: 'ï»¿"Date"'


texts: Scala mllib 2.2.0 k means. Number of runs

texts:
As per the documentation here, the runs parameter does nothing.
I have looked at the actual Scala code for k means and it appears that the train method only does one start. There is no way to change this.

QUESTION:
Am I correct that if I want 5 starts I need to write my own code that calls train 5 times (or until some criteria has been met)?


texts: Where is the Backward function defined in PyTorch?

texts:
This might sound a little basic but while running the code below, I wanted to see the source code of the backward function:
import torch.nn as nn

[...]
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
loss.backward()

So I went to the PyTorch GitHub and found the CrossEntropyLoss class, but without any backward function defined. Moving up, CrossEntropyLoss extends _WeightedLoss >> _Loss >> Module then still nothing.
So, where is the backward function defined?


texts: Resource-unintensive (low complexity) methods for large-scale unsupervised clustering?

texts:
I'm working on an issue where I need to cluster user types on a scale in an unsupervised manner. I've been looking at the basics like KNN and K-means etc., but I found it hard to scale, as these methods are quite resource-intensive.
What are some highly scalable clustering methods that have a low complexity ("big O")?


texts: Proper/Possible methods for extracting unstructured data from websites

texts:
I'm working in Python, using Scrapy, and NLTK to try to understand how I can extract data from college websites.

My scraper can navigate through the university websites and find their tuition fees pages perfectly , but when trying to extract specific fees like :

Resident
Non Resident
Per Credit Hour
Per Semester

I'm running into trouble due to the data being so unstructured from site to site.

I've tried using NLTK to parse data based on parts of speech tags and regex chunking to try to extract sentences such as "tuition cost for resident: $12,500" but colleges can display this data in a number of ways.

Here is my question:

Are there any better ideas/methodologies that I should be looking into that can help me with extracting this type of data?


texts: Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?

texts:
ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.

Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. Backpropagation in first-order methods requires first-order derivative. Hence $x$ is derived to $1$.

But if we use second-order methods, would ReLU's derivative be $0$? Because $x$ is derived to $1$ and is derived again to $0$. Would it be an error? For example, with Newton's method, you'll be dividing by $0$. (I don't really understand Hessian-free optimization, yet. IIRC, it's a matter of using an approximate Hessian instead of the real one).

What is the effect of this $h''=0$? Can we still train the neural network with ReLU with second-order methods? Or would it be non-trainable/error (nan/infinity)?

For clarity, this is ReLU as $f(x)$:

$f(x) =$ \begin{array}{rcl}
 0 & \mbox{for} & x < 0\\
 x & \mbox{for} & x \ge 0\end{array}

$f'(x) =$ \begin{array}{rcl}
 0 & \mbox{for} & x < 0\\
 1 & \mbox{for} & x \ge 0\end{array}

$f''(x) = 0$


texts: Interpreting statistical formulas using python (mu + sigma * data)

texts:
I am working with data using python. I tried to plot a histogram using mathplotlib.

I found a few examples on how to analyze data to get more accurate graphs.

So I coded:

# Call csvfile
csvdata = pandas.read_csv('data/education_data/' + filename + '.csv', low_memory=False, encoding='latin-1')

# Convert amount field into int()
def str_to_int(mainList):
    for item in mainList:
        newList = [(int(item.replace('$', '').replace(',', '')) / (1000000)) for item in mainList]
    return newList

# Convert str to int using str_to_int() function
csvdata['str_costo'] = str_to_int(csvdata['Cuantía'])
sigma = numpy.std(csvdata['str_costo'])
mean = numpy.mean(csvdata['str_costo'])
x = mean + sigma * csvdata['str_costo']

But, I saw one step in particular:

In[0] x = mean + sigma * csvdata['str_costo']
Out[0] 1.371822e+08, 1.608953e+08, etc...

All values from csvdata['str_costo'] are integers.

My question is how to interpret x values.

Theoretically what they mean?


texts: What is the simplest optimization algorithm for a multi-parameter closed system?

texts:
I'm working on a minimization problem for a wireless communication link. I want to minimize the total bit error rate (BER) of a closed system. My problem is that the signal has multiple (50) subcarriers. Each of them having 2 parameters impacting the bit-error rate. I'm calling them a and b. Both have a minimum and maximum value, as well as a fixed increment between the potential values.

My question is: what is the simplest way to minimize the BER in such a scenario ?

Just a thing i need to point out: With a brute force loop algorithm setting the same a and b for all subcarriers (a_u and b_u), I attain a minimum BER that is relatively close to the global minimum. That is to say, when i start optimizing from these newly found a_u and b_u, the global minimum is always found by increasing a for the first few subcarriers and decreasing it for the few last subcarriers. And a similar scenario occur to b except that my issue is that i have to optimize both at the same time, which I think is sub-optimal through a brute force loop.

So my thought was to apply an optimization algorithm from this starting point (a_u and b_u) that make sense in this scenario.


texts: How to perform cropping on large file dataset using ImageGenerators?

texts:
I have a large dataset of RGB images with 220x220x3. I want to crop the region of interest from each of them. For that, I have defined my own function. Im trying to use this with DataImageGenerators but it's only resizing rather than cropping.
def crop(image):

    x=0
    h=110
    print(image.shape)
    return image[x:x+h,:,:]

IMAGES_DIR1='/home/imtiaz/Desktop/Medium/Article1/Image_preprocessing/All towns/All'

test_datagen = ImageDataGenerator(rescale = 1./255, preprocessing_function=crop)

test_generator = test_datagen.flow_from_dataframe(
        dataframe=df_test,
        directory=IMAGES_DIR1,
        x_col="Images",
        y_col="steer",
        target_size=(110, 220),
        class_mode='other')

And when I check those images using

from keras.preprocessing.image import array_to_img
array_to_img(x[0])

When I display the image it isn't the cropped one. What am I doing wrong ?


texts: Are there any methods for finding the value of variable which has significant influence on response?

texts: I have a dataset which has 5 variables and 1 response. The variables are discrete. I want to find the key variable and its value which leads to a significant increase or decrease to the response.


texts: How to measure variance in a classification dataset?

texts:
I have a dataset that contains 20 predictor variables (both categorical and numeric) and one target variable (2 classes - Good and Bad). But, there are only 23 observations in the dataset. While I wait to receive significantly more observations, what tests / models can I perform on the available dataset to understand the variance between the good and bad cases, and to understand the variance within the cases classified as 'good'?

Ideally, for the data to make sense, I would want the variance within the good cases to be low, and the variance between the good and bad cases to be high.

Would multivariate analysis of variance (MANOVA) work in this case?


texts: is Batch Norm a little bit stochastic by default?

texts:
Using full-batch gradient descent, stacking 100 layers and using alpha 0.0001 results in steadily decreasing error.

However, after I implemented Batch Norm, the same scenario results in fluctuations. My implementation was verified by several people, so now I am wondering - why batch norm adds this stochastisity effect?

In fact, I am no longer able to stack 100+ layers, only approximatelly 10 layer before stochasticity becomes very apparent and hard to control.

What's more interesting -it seems to get worse with smaller learning rate! 0.4 is good, and 0.0001 results in smaller updates (as expected) but larger relative fluctuation (caught me by surprise).

Why is it so?

Edit: Just tried 100 layers (each 63 neurons). It's very noisy, but I am able to more-or-less steadily reduce the error. It's less stochastic if I set learning rate to 10 ("ten", lol!) and will get only noisier if it's say 0.5 which is incredibly weird

Of course, this learning rate is so high that the error does occasionally snap to a high value, but seems to work with 100 layers...

Notice - I am using full-batch gradient descent, with 50 elements

Edit:

Using 100 LSTM units, stacked like a pillar onto each other.

Number of timesteps is 50. Feature size 50 (lstm state has dimension of 50), because my sentence has 50 distinct characters, each character is encountered only once per epoch.

Performing backpropagation after 50 timesteps Using vanilla gradient descent, all fancy things like accelerated momentum or L2 norm, dropout are disabled.

I am sure there is no error, but interested if anyone can tell why Batch Norm has this stochastic property, if that's common with people.


texts: Association rule mining and Orange3 package Python

texts: how i use Orange3 package for association rule mining ,in Orange 2.7 version Orange.associate.AssociationRulesSparseInducer() method is present but its not available in Orange3 ..


texts: Pandas df - FacetGrid - Need help with plotting

texts:
Let's say I have a df with the below columns

Feed     ItemName
----     --------
FeedA      ItemA
FeedA      ItemB
FeedA      ItemC
FeedA      ItemA
FeedA      ItemB
FeedA      ItemA
FeedA      ItemC
FeedA      ItemC
FeedA      ItemC
FeedA      ItemA
FeedA      ItemB
FeedB      ItemC
FeedB      ItemB
FeedB      ItemB
FeedB      ItemB
FeedB      ItemC
FeedB      ItemA
FeedB      ItemA
    .....
    .....

    .....

I want to get the count of each ItemName grouped by Feed and build a Facet grid (for each Feed)

Whilst I tried
df.groupby(['Feed','ItemName'])['ItemName'].count()
to the get the count split by ItemName & grouped by Feed, but am not sure on how plot that as graph.


texts: How to train BERT (multi label) on imbalanced dataset for search query category classification

texts:
I have a dataset of 2 million search queries relative to 7000 categories. same query could have multiple categories. Aim is to predict category/categories for query with confidence score.
I tried using bert multi label with all categories in single column with text. so for queries with multiple categories the same query phrase is repeated in multiple rows with categories. accuracy of model is very low around 40%. i believe following could be issues
1.imbalanced data from 100 queries per category to 80k queries
2.maybe 7000 categories should be rearranged as column and each marked as yes/no for the respective query.
now 1st issues is a little manageable. will the second one be any better or there could be a  approach to follow?


texts: How to make a neural network generalizes better?

texts: I designed a neural network model with large number of output predicted by softmax function. However, I want categorize all the outputs into 5 outputs without modifying the architecture of other layers. The model performs well in the first case but when I decrease the number of output it loses accuracy and get a bad generalization. My question is: Is there a method to make my model performs well even if there is just 5 outputs? for example: adding dropout layer before output layer, using other activation function, etc.


texts: Could anyone explain these terms, "input space", "feature space", "sample space", "hypothesis space", "parameter space" with a concrete example?

texts:
People use these terms "input space", "feature space", "sample space", "hypothesis space", "parameter space" in machine learning.

Could anyone explain these terms with a concrete example, such as sklearn MNIST dataset?, which has 1797 Samples, 10 Classes, 8*8 Dimensionality and 17 Features.

Please do NOT talk about in general.

For example, in this particular case, is the feature space a set of 17 elements {0, 1, ..., 16}?


texts: Equation for likelihood in logistic regression

texts:
I read the likelihood is defined in logistic regression as the probability
$$
L(w) = P(y|x, w) = \prod P(y^i| x^i,w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}
$$
and the log of the last equation is:

$$
log(L(w)) = \sum y^i log((\sigma(z^i)) + (1-y^i)log(1-\sigma(z^i))
$$

I understand (independent probabilities)
$$
L(w) = P(y|x, w) = \prod P(y^i| x^i,w)
$$
and I understand that from
$$
L(w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}
$$
the log is (basic log properties)
$$
log(L(w)) = \sum y^i log((\sigma(z^i)) + (1-y^i)log(1-\sigma(z^i))
$$
However, how do I get
$$
\prod P(y^i| x^i,w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}
$$
This basically means that
$$
P(y| x,w) = (\sigma(z))^{y}(1-\sigma(z))^{(1-y)}
$$
and I just don't see that.


texts: How would you a apply a cnn to do age estimation on static images?

texts: After doing some reading on age estimation using the IMDB wiki dataset I wanted to try it out myself on a smaller scale but I dont quite understand the application of the CNN. Any clarification would be great.


texts: Cross-Validation: Repeated K-Fold/Group K-Fold

texts:
Repeated K-Fold vs Group K-Fold

As per my understanding from sklearn docs

Repeated K-Fold:

RepeatedKFold repeats K-Fold n times. It can be used when one requires to run KFold n times, producing different splits in each repetition.

Repeated Stratified K-Fold cross validator:

Repeats Stratified K-Fold n times with different randomization in each repetition.

Group K-Fold:

GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing and training sets.

  Can somebody explain in-detail, When would one use Repeated K-Fold over Group k-fold?
  What are the advantages/disadvantages of using Repeated K-Fold over Group k-fold?


texts: How to do kfold cross-validation for multi-input models

texts:
The model is as below:

inputs_1 = keras.Input(shape=(10081,1))

layer1 = Conv1D(64,14)(inputs_1)
layer2 = layers.MaxPool1D(5)(layer1)
layer3 = Conv1D(64, 14)(layer2)
layer4 = layers.GlobalMaxPooling1D()(layer3)

inputs_2 = keras.Input(shape=(85,))
layer5 = layers.concatenate([layer4, inputs_2])
layer6 = Dense(128, activation='relu')(layer5)
layer7 = Dense(2, activation='softmax')(layer6)

model_2 = keras.models.Model(inputs = [inputs_1, inputs_2], output = [layer7])

X_train, X_test, y_train, y_test = train_test_split(df.iloc[:,0:10166], df[['Result_cat','Result_cat1']].values, test_size=0.2)
X_train = X_train.to_numpy()
X_train = X_train.reshape([X_train.shape[0], X_train.shape[1], 1])
X_train_1 = X_train[:,0:10081,:]
X_train_2 = X_train[:,10081:10166,:].reshape(736,85)

X_test = X_test.to_numpy()
X_test = X_test.reshape([X_test.shape[0], X_test.shape[1], 1])
X_test_1 = X_test[:,0:10081,:]
X_test_2 = X_test[:,10081:10166,:].reshape(185,85)

adam = keras.optimizers.Adam(lr = 0.0005)
model_2.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['acc'])
history = model_2.fit([X_train_1,X_train_2], y_train, epochs = 120, batch_size = 256, validation_split = 0.2, callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)])

Questions:

1) The data is 921rows x 10166columns. Each row is an observation(first 10080 columns being a time series with remaining columns being other statistics features). According to the model, is the input data split into inputs_1 and inputs_2 randomly?

2) I am thinking about doing a kfold cross-validation and splitting the input data into inputs_1 and inputs_2. What is a good way to do this? Thanks