texts: Prediction questions related to the dataset

texts:
I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably.

Description: It is the response of survey and results regarding usage of services for financial transactions using traditional methods or mobiles as a medium in a developing country. The study was conducted to understand the usage of technology for financial services by men and women in developing countries to find out how comfortable they are to use technology for this purpose. A dictionary is provided with the dataset which you may refer to column details. The dataset is big and contains many missing values. In the dataset, for male they have used 0 and for female are using 1. The purpose of this problem is that you will be able to understand how to process real data for valuable information extraction.

Dataset dictionary: https://1drv.ms/x/s!ArGTGzC7esWNa25KnJACZTXuGPw

Dataset: https://1drv.ms/u/s!ArGTGzC7esWNbCQevyUWRWhgrl8

The tasks given to my friend were:

1) Preprocess the data

2) Which feature engineering techniques you used?

3) predict the gender of the survey respondent.

4) Predict that respondent is female with a certain probability. 5) Find interesting patterns that are associated with a specific gender

1) I want to know what preprocess can I do in the data. I can remove missing and duplicate values, but for missing values there is a problem. If the survey respondent chose other in the religion column (96 value in column DG3A), they specified the name in the next column. But if they gave a response such as 1 (which is for Christianity) in the DG3A column then the value in the next column would be empty. How would I cater for these missing values (which aren't really missing instead needed not to be filled)?

2) I am not sure which ones should I use and how.

Which anyone can also guide me which columns to use and how for the remaining parts it would be great. :)


texts:
I've been looking around for pretrained models dedicated to handwritten OCR. So far I've found very little. Could you please share, if you know any?
I find tesseract hard to parse anything that isn't arial and perfectly captured.


texts: What algorithm is used to extract keywords from unstructured texts?

texts:
I'm using StanfordNLP (java pakage) for twitter sentiment analysis. I want to extract keywords from the tweets, classify the keywords as positive, negative, or neutral, and use those counts to classify the polarity of the tweet.

How can this be done in Java? Please share few examples. What is the most used algorithm for keyword extraction?


texts: What is the difference between additive and multiplicative attention?

texts:
This paper (https://arxiv.org/abs/1804.03999) implements additive addition. I think the attention module used in this paper (https://arxiv.org/abs/1805.08318) is an example of multiplicative attention, but I am not entirely sure. Can anyone please elaborate on this matter?
Also, the first paper mentions additive attention is more computationally expensive, but I am having trouble understanding how. Any insight on this would be highly appreciated. Thank you.


texts: Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.)

texts:
I am trying to do a correlation study about personality traits as described in Hofstede's :https://www.hofstede-insights.com/product/compare-countries/  . I would like to have the values described in the bar charts numerically into, say an Excel or pandas file. Is there a way of scraping or using an API that would turn the bar chart values into number values associated to each country? I looked at similar questions but the closest I found was transforming categorical data given in a file format and did not involve scraping/APIs to change categorical into numerical.
Thanks.


texts:
I'm working on a Sentiment Analysis task using TF-IDF to build my features and SVC as the classifier.
My goal is to make my model to classify the sentiment of all my dataset. I already designed my model with training and test data, splitting my dataset into training and test sets, but now I want my model to give the labels for each element of the dataset.
For that, I made the following steps

Applied the TF-IDF fit_transform() to my train dataset
Applied the TF-IDF transform() to my entire dataset
Trained a SVC with my train dataset
Predicted the sentiment of my entire dataset
Got the classifications and the probabilities for each class

from sklearn.svm import SVC
from sklearn import metrics

Tfidf_X_train = Tfidf.fit_transform(X_train['text']).toarray()

Tfidf_df  = Tfidf.transform(df_preproc['text']).toarray()

svc_classifier = SVC(probability=True)
svc_classifier.fit(Tfidf_X_train, y_train)
y_pred = svc_classifier.predict(Tfidf_df)
class_probabilities = svc_classifier.predict_proba(Tfidf_df)

The problem is when I look what class_probabilities or y_pred returns it gives me duplicated lines, as if it classified the same text more than once, even I knowing that each text is unique in the dataset. Like the following example:

Where am I doing wrong?


texts: Need explanation of a matrix multiplication

texts:
I'm reading the Deep Learning book by MIT.
On the page 172, there's a part like this:
$$
f^{(1)}(x)=h=W^Tx  \tag{1}
$$
$$
f^{(2)}(h)=h^Tw \tag{2}
$$
Substitute (1) into (2), they got:
$$
f(x)=w^TW^Tx
$$
Since I'm not so familiar with linear algebra stuff, I infer that something like below is valid:
$$
A^TB=B^TA \tag{3}
$$
So what is the property of (3) is called?


texts: Using HashingVectorizer for text vectorization

texts:
Here is the sample data I have:

Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)

Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)

Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)

Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)

I started by putting the Tags into a dataframe df and the Label into a separate dataframe df_label.
Then used a HashingVectorizer to prepare the text for processing by ML models (I want to hash the strings into a unique numerical value so that the ML Models can train on it)

vectorizer = HashingVectorizer()

X_train = vectorizer.transform(df)

y_train = vectorizer.transform(df_label)

clf = RandomForestClassifier(n_jobs=2, random_state=0)

clf.fit(X_train, y_train)

When I execute this, I get: ValueError: Unknown label type: 'unknown' on y_train.

I am new to both Python and ML and I am not sure whether the problem is with my basic logic or whether it is a trivial implementation issue. Appreciate your insight and support.


texts: Can Machine Learning be applied in software developement

texts:
I'm from programming background. I'm now learning Analytics. I'm learning concepts from basic statistics to model building like linear regression, logistic regression, time-series analysis, etc.,

As my previous experience is completely on programming, I would like to do some analysis on the data which programmer has.

Say, Lets have the details below(I'm using SVN repository)

personname, code check-in date, file checked-in, number of times checkedin, branch, check-in date and time, build version, Number of defects, defect date, file that has defect, build version, defect fix date, defect fix hours, (please feel free to add/remove how many ever variables needed)

I Just need a trigger/ starting point on what can be done with these data. can I bring any insights with this data.

or can you provide any links that has information about similar type of work done.


texts:
I am working on a dataset with physical measurements taken daily (weight, bmi, etc...) and I am working through the process to graphically represent it. I think it is worth noting that every day has a corresponding row, but if no measurements were taken, the values are the same as the day before.

Here is an example of the trend I am trying to manage:

Date, Weight, BMI
1/1/2016, 155.1, 21.9
1/2/2016, 155.1, 21.9
1/3/2016, 155.1, 21.9
--continued for several weeks--
3/1/2016, 170.2, 25.0
3/2/2016, 170.1, 25.0

edit: I should clarify that these repeated values are how the data is put together for missing values. The days that are repeated are days when no measurement was taken

If the value stays the same for an extended period, should there be a gap in any graphical representation? Should I keep the numbers as they were (155.1 & 21.9) until the next measurement was taken, or should the numbers increase over that time to "bridge the gap" - meaning they would increase by the difference in measurements divided by the number of days?

It feels like I should be increasing the value over time to account for what would happen in reality, but I don't know if this would have negative implications on the data.


texts:
this might be simple  but I need to know how to choose a best algorithm based on a scenario.

I have a dataset. The target class is, let's say color, this color attribute can have upto 5 values (red, yellow, blue, green, pink)

So I have:

1.) 1 target class - color

2.) Multiple values for that target class - (red, yellow, blue, green, pink)

As this is a labelled dataset with known classes, I can use a Supervised Learning Algorithm

But I don't know which algorithm would be the best. It should be a multi-class classigication? Or multi-labeled ? Can someone help me to find the best algorithm?

note: I know choosing an algorithm depends on many factors (such as its acuracy), but at a glance, what type of algorithms should I try?

Thank you


texts: PCA and maintaining relationship with target variable

texts:
I'm rather new to PCA and was hoping to have some confusion cleared up. Lets say for example we have a feature matrix that's nx100 and I want to get it down to something a bit smaller, p-dimensions, without losing too much variance.

After applying PCA and receiving and new feature matrix nxp, I would use x_reduced to predict some target variable y.

My question is, after the transformation, the new reduced feature matrix has been rotated by the eigenvectors and is sitting on a new basis. Yet, our y has not changed relative to X_reduced.

I'm unsure about how y_original and x_reduced can be used for training since y has not changed with respect to x_reduced.

Is there a way to correct for this or am I not thinking about it correctly?


texts:
R has many libraries which are aimed at Data Analysis (e.g. JAGS, BUGS, ARULES etc..), and is mentioned in popular textbooks such as: J.Krusche, Doing Bayesian Data Analysis; B.Lantz, "Machine Learning with R".

I've seen a guideline of 5TB for a dataset to be considered as Big Data.

My question is: Is R suitable for the amount of Data typically seen in Big Data problems?
Are there strategies to be employed when using R with this size of dataset?


texts: What is the meaning of hand crafted features in computer vision problems?

texts: Are these the features which are manually labelled by humans? or  Is there any technique for obtaining these features. Is this related to learned features?


texts: Pandas, make the bars of a bar graph have equal widths

texts:
What do I need to change so that ax1 uses the same width for bars as ax2?

import pandas as pd
import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(2)

pd.Series([1]).plot(kind='bar', width=0.5, ax=ax1)
pd.Series([1, 2, 3]).plot(kind='bar', width=0.5, ax=ax2)

plt.show()


texts: There are 2 figures explaining transposed convolution. Which one is correct?

texts:
I have been struggling to understand transposed convolution. When I search for "transposed convolution", there are 2 figures explaining transposed convolution in which I think are not equivalent (or are they?).

Figure 1. Transposed Convolution as explained here

Figure 2. Transposed Convolution as explained here

Additionally, in Stanford's CS231N, transposed convolution was explained as in the Figure 1 (here, slide 52). Meanwhile, another online lecture from TUM's DVL explained transposed convolution as in the Figure 2 (here).


texts: How do I choose a laptop if I'm interested in learning and applying data science?

texts:
I just got started with learning data science, and I was wondering what type of laptop I'd need to buy. I understand that this might come probably a little too early -- but my current laptop absolutely needs to be replaced and I don't want to buy a new laptop only to find out about a year later that I can't apply what I've learnt.

So, what are the minimum specs I should be looking out for? I've read about Amazon's AWS that allows us to create and access virtual machines using the cloud -- is that a good idea, especially considering that I'm only starting out?

I won't be using a laptop for any other intensive activities such as gaming or video-editing. Should I get a high-end laptop or stick to a basic model and just use cloud based services for my data science needs? If the latter (buying a basic model) is a better choice, does that also need certain basic hardware requirements?


texts:
For the task of detecting an object of any color, if we train our CNN only with images that only contain that object in one or two colors, will the accuracy of our model's predictions be affected for images containing that object with different colors than the ones used on training?
If so, what is the best way to attain color invariance? (Examples and references to credible sources would be much appreciated.)


texts: What are some methods for clustering individuals into distinct groups based on Features?

texts:
I started with a dataset that contained many dimensions for individuals (each id is a separate individual), and extracted three Features/Attributes columns for each individual Id.

My goal is to bucket these individuals into two or three distinct groups, based on these features, to see if I can identify distinct separations between certain groups.

I was wondering if anyone has any recommendations, or algorithms (preferably in Python) that would cluster these individuals into distinct groups?  I do not have these Individuals classified, so it's an unsupervised clustering problem.  I was thinking that K-Means might be a good option, or something similar to PCA which could reduce my dimensionality and provide insight into the Features that seem to separate the groups of individuals into distinct groups.

Thanks for looking!

Note: The data shown below was artificially generated to illustrate my question.

Appendix: Reference Data:

Pattern,Feature A,Feature B,Feature C

6,2.18,0.13,8.00
9,9.31,3.67,6.58
11,0.89,1.83,4.33
13,9.73,9.50,1.59
23,0.51,6.49,0.09
26,9.04,3.42,3.90
27,8.35,9.18,3.40
28,3.04,5.63,6.88
32,9.78,6.52,8.50
43,4.11,3.36,1.83
49,9.57,1.52,7.09
51,1.13,9.98,9.42
53,6.22,1.37,7.07
62,8.79,3.03,7.52
63,6.27,7.29,0.98
71,4.64,0.06,6.55
73,1.34,9.32,5.15
83,4.53,3.85,2.04
84,9.48,9.71,3.23
86,3.80,3.00,0.76
88,1.73,0.64,9.96


texts:
How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )?
I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear training data that I can put in my code at tensorflow or keras! How we can do this?


texts: NoSQL engine/service recommendation for geolocation data

texts:
First of all, I am new in this field we call big data, so my questions may be naive.

In order to build an application, which deals with geolocation data, which could be : latitude and longitude coordinates and Geography SQL Server column types.

I need to have the following elements made easy:

Scalability : be prepared to receive huge amount of data, adding servers to the system have to be easy
Proximity requests : in example, how much points are in a circle (at meter scale).
Data must be accessible rapidly after being written.

I've been looking around for existing solutions, which are "Hadoop friendly" (Hortonworks, Cloudera) and available DBMS, like Cassandra.
I have found some interesting information, but I still think it's hard to decide, which one to choose.
It also need drivers for NodeJS & .NET (Hadoop with Cassandra seem to be OK with that).
I've also looked around the MongoDB ecosystem, but, again, I feel that it is hard to know where to look at. By (little) experience with Mongoose, MongoDB can be disqualified by the third point because data writes are slow. But my model could certainly be improved.

Do any of you have some recent experiences, manipulating massive amount of geolocation data? I would appreciate sharing them here as well as any quality and recent literature on the subject.


texts: Tensorflow: Unable to Save checkpoint after every 2 global steps during training the SSD model object detection

texts:
python models/object_detection/train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_inception_v2_coco.config

INFO:tensorflow:Restoring parameters from /home/rahul/Downloads/ssd_inception_v2_coco_2018_01_28/model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path training/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:global step 1: loss = 14.1708 (73.929 sec/step)
INFO:tensorflow:global step 2: loss = 13.3957 (26.779 sec/step)
INFO:tensorflow:global_step/sec: 0.0207936
INFO:tensorflow:Recording summary at step 2.
INFO:tensorflow:global step 3: loss = 13.2996 (34.331 sec/step)
INFO:tensorflow:global step 4: loss = 12.6129 (27.737 sec/step)
INFO:tensorflow:global step 5: loss = 12.0835 (28.638 sec/step)
INFO:tensorflow:global step 6: loss = 11.9736 (29.535 sec/step)
INFO:tensorflow:global_step/sec: 0.0333812
INFO:tensorflow:Recording summary at step 6.
INFO:tensorflow:global step 7: loss = 11.3325 (35.411 sec/step)
INFO:tensorflow:global step 8: loss = 10.9632 (28.500 sec/step)
INFO:tensorflow:global step 9: loss = 10.8758 (27.419 sec/step)
INFO:tensorflow:global step 10: loss = 11.1301 (25.544 sec/step)
INFO:tensorflow:Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
rahul@rahul-pc:~/crack_detection_ssd_inception$

For above it is saving checkpoint at 0th & 10th step but I want to save checkpoint after every 2 global steps for which i re-trained my model only for 10 num_steps (this is only for testing).
from where we can modify the checkpoint saving?

I tried to change save_checkpoints_steps, save_checkpoints_secs in object_detection/model_main.py & object_detection/model_lib_test.py


texts: Text classification- What to do when train and test data have different features

texts:
I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech.

So as general thumb rule i preprocessed my data. create term document frequency and After removing sparse terms i divide my data into train and test.
I train my model using random forest and logistic regression and it worked fine.

set.seed(123)
tweetRand = randomForest(label ~ ., data = train_sparse, importance=TRUE, nTree=500 )
randPridct = predict(tweetRand, newdata = test_sparse)
table(test_sparse$label,randPridct >=0.5)

Its is working fine on test data which divided from raw content.
But when i am running it on a new unseen data it is throwing an exception.

> predicrRand_test=predict(tweetRand, newdata=sparse_4testing)
Error in eval(predvars, data, env) : object 'run' not found

My understanding is that 'run' is a feature present in training but not in unseen test data and during my model training 'run' was included in tdm. In preprocessing of test , run was not in test tdm.

SO how should i deal with these situation. I am new to data science. Please help.


texts: Model ensemble with Spark or Scikit Learn

texts:
I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators.

Here is a reproducible of what I would like my model to do with the Spark api

from sklearn.datasets import load_diabetes
import pandas as  pd
import pyspark
from pyspark.ml.feature import VectorAssembler, SQLTransformer
from pyspark.ml.classification import LogisticRegression
from pyspark.ml import Pipeline
# Query diabetes data
diab = load_diabetes()
df = pd.DataFrame(diab.data, columns=diab.feature_names)
df['is_male'] = df.sex > 0
df.drop('sex', inplace=True, axis=1)
df['label'] = diab.target
# Model made with Spark

spark = pyspark.sql.SparkSession.builder.master('local[10]').appName('A random spark context').getOrCreate()
def create_gender_model_male():
    return Pipeline(stages=[SQLTransformer(statement='SELECT * FROM __THIS__ WHERE is_male'),
                                VectorAssembler(inputCols=['age', 'bmi', 'bp', 's1'],outputCol='features'),
                                LogisticRegression(featuresCol='features', labelCol='label', maxIter=100,
                                                   elasticNetParam=1, regParam=0.)
                                ])
def create_gender_model_female():
    return Pipeline(stages=[SQLTransformer(statement='SELECT * FROM __THIS__ WHERE not is_male'),
                                VectorAssembler(inputCols=['age', 'bmi', 'bp', 's1'],outputCol='features'),
                                LogisticRegression(featuresCol='features', labelCol='label', maxIter=100,
                                                   elasticNetParam=1, regParam=0.)
                                ])
df = spark.createDataFrame(df)
class MixedModel():
    def __init__(self):
        self.models = {'male': create_gender_model_male(), 'female': create_gender_model_female()}
        self.fitted_models = {'male': None, 'female': None}
    def fit(self, df):
        self.fitted_models['male'] = self.models['male'].fit(df)
        self.fitted_models['female'] = self.models['female'].fit(df)
    def predict(self, df):
        return self.fitted_models['male'].transform(df).union(self.fitted_models['female'].transform(df))
mm = MixedModel()
mm.fit(df)
mm.transform(df)

Here, for example I have one logistic regression per sex but I would also like to be able to have prediction with a tree for males and prediction with Logistic regression for females if I want.

In a perfect world there would be a function:

ModelAggregation(('is_male is true, male, model_for_male), ('is_male is false', model_for_female)))

which would return me an object like my model aggregation


texts: Does stronger regularization always improve performance on testing set?

texts:
I am using the Sklearn logistic regression function to do a binary classification task on texts.
I did the task using three different inputs: Bag-Of-Words, TF-IDF, Doc2vec embeddings.
The question is that for the Bag-Of-Words, stronger regularization improves the performance on testing set.
However, for TF-IDF and Doc2vec embedding, stronger regularization doesn't improve the performance on testing set. Actually the performance downgrades with stronger regularization.
Why this is the case?


texts:
Having an issue with GridSeachCV. I am using sklearn 0.24.1. It is has been stuck in "Fitting 5 folds for each of 864 candidates, totalling 4320 fits" for a long time.
my dataset only contains about 24k rows. Here is the code.
n_estimators = [int(x) for x in np.linspace(start = 10, stop = 200, num = 2)]
max_features = ['auto', 'sqrt']
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
min_samples_split = [2, 5, 10]
min_samples_leaf = [1, 2, 4]
bootstrap = [True, False]

grid_rf = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}

rf = RandomForestClassifier()
grid = GridSearchCV(rf, grid_rf, cv = 5,verbose = True, n_jobs = -1)
grid.fit(xtrain,ytrain)
params = grid.best_params_
print(params)

I have set verbose to true as well but it's not showing up. It usually works pretty quickly, but I don't know why it's not loading this time. Any ideas?


texts:
I am having difficulty finding where my error is while building deep learning models, but I typically have issues when setting the input layer input shape.

This is my model:

model = Sequential([
Dense(32, activation='relu', input_shape=(1461, 75)),
Dense(32, activation='relu'),
Dense(ytrain.size),])

It is returning the following error:

 ValueError: Error when checking input: expected dense_1_input to have 3

 dimensions, but got array with shape (1461, 75)

The array is the training set from the kaggle housing price competition and my dataset has 75 columns and 1461 rows. My array is 2 dimensional, so why are 3 dimensions expected? I have tried adding a redundant 3rd dimension of 1 or flattening the array before the first dense layer but the error simply becomes:

ValueError: Input 0 is incompatible with layer flatten_1: expected

min_ndim=3, found ndim=2

How do you determine what the input size should be and why do the dimensions it expects seem so arbitrary?

For reference, I attached the rest of my code:

xtrain = pd.read_csv("pricetrain.csv")
test = pd.read_csv("pricetest.csv")
xtrain.fillna(xtrain.mean(), inplace=True)
xtrain.drop(["Alley"], axis=1, inplace=True)
xtrain.drop(["PoolQC"], axis=1, inplace=True)
xtrain.drop(["Fence"], axis=1, inplace=True)
xtrain.drop(["MiscFeature"], axis=1, inplace=True)
xtrain.drop(["PoolArea"], axis=1, inplace=True)
columns = list(xtrain)
for i in columns:
    if xtrain[i].dtypes == 'object':
        xtrain[i] = pd.Categorical(pd.factorize(xtrain[i])[0])
from sklearn import preprocessing

le = preprocessing.LabelEncoder()
for i in columns:
    if xtrain[i].dtypes == 'object':
        xtrain[i] = le.fit_transform(xtrain[i])
ytrain = xtrain["SalePrice"]
xtrain.drop(["SalePrice"], axis=1, inplace=True)
ytrain = ytrain.values
xtrain = xtrain.values
ytrain.astype("float32")

size = xtrain.size
print(ytrain)
model = Sequential(
    [Flatten(),
     Dense(32, activation='relu', input_shape=(109575,)),
     Dense(32, activation='relu'),
     Dense(ytrain.size),
     ])
model.compile(loss='mse', optimizer='adam')
model.fit(xtrain, ytrain, epochs=10, verbose=1)

Any advice would be incredibly helpful!

Thank you.


texts: What does NIST information weights refer to?

texts:
NIST is a metric used to measure the goodness of translation.

In the paper, Doddington (2002) introduce the notion of "Information weights"

  Information weights were computed using N-gram counts over the
  set of reference translations, according to the following equation:

Which Owczarzak (2008) describes as

  Here the precision is modified by the information weights Info(w1 . . . wn), computed on the basis of the frequency of a given n-gram in relation to the frequency of a ‘parent’ (n-1)-gram, in accordance with the equation (2.6)

There's an implementation of the NIST score on https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v13a.pl

But we can't seem to figure out how should the information weights be computed on https://github.com/nltk/nltk/pull/1840

Does anyone know the details of how this information weights work?

I've tried looking at the mteval-v13a.pl script but I'm not understanding how it works and relates to the formula above.


texts: Is there a way to use Plotly as an annotation tool, for labelling time-series for instance?

texts:
I have been tasked to create a tool aimed at labelling sections and/or precise data points of a biomedical time-series. Our main framework is written in Python.

I would like to know whether it is possible to do it with Plotly, and if so, would someone mind sharing their experience?

Kind regards,
Bertrand


texts: Is it good practice to remove the numeric values from the text data during preprocessing?

texts:
Im doing preprocessing on a text dataset. I have certain numerics in it like:

date(1st July)
year(2019)
tentative values (3-5 years/ 10+ advantages).
unique values (room no 31/ user rank 45)
percentage(100%)

Is it recommended to discard this numerics before creating a vectorizer(bow/tf-idf) for any model(classification/regression) development?
Any quick help on this is much appreciated. Thank you


texts: Does fine-tuning of transferred layers perform better than frozen transferred layers?

texts: I recently learned concepts of transfer learning. Is it necessarily true that fine-tuning of transferred layers perform better than frozen transferred layer? why?


texts: How to go about cooling load forecast for a district level dataset?

texts: I am trying to build a high accuracy cooling load forecast for a district-level dataset over various time horizons. My data consists of a time series of cooling load and consecutive weather data. I added a bunch of other features such as an hour of the day, day of the week, holiday or not, etc. My question is, what are some current models for the prediction of the time series load dataset like mine? I gather LSTM is the way to go. However, given the features I have identified, are there other models that are more accurate for such a forecast? If I chose to go the NN fitting route, what are some other features I can add?


texts: Can you combine two xgboost models into one?

texts: If you have built two different xgbost models, with say 100 trees each, is it possible to combine into an xgboost model with 200 trees?


texts: Gradient flow through concatenation operation

texts:
I need help in understanding the gradient flow through a concatenation operation.

I'm implementing a network (mostly a CNN) which has a concatenation operation (in pytorch). The network is defined such that the responses of passing two different images through a CNN are concatenated and passed through another CNN and the training is done end to end.

Since the first CNN is shared between both of the inputs to the concatenation, I was wondering how the gradients should be distributed through the concatenation operation during backprop? I'm not an expert on backprop and this is the first time I'm tinkering with a custom backward implementation so any pointers would be helpful.

I can provide more details if you guys need it.


texts: Prohibitive size of random forest when saved to disk

texts:
When saved to disk using cPickle: https://stackoverflow.com/questions/20662023/save-python-random-forest-model-to-file, my random forest is 6.57 GB.

with open('rforest.cpickle', 'wb') as f:
    cPickle.dump(rforest, f)

I want to use the forest itself to make predictions via a python API hosted on Heroku -- of course, that file size is unacceptable.

Why is the file size so large? There are 500 trees in the ensemble -- all I want to save are the completed trees themselves, since they will be used as prediction. Is it the actual nodes and edges that compose each of the 500 trees that requires nearly 7 GB of space on disk?

I used scikitlearn's randomforestregressor:

def buildForest(self, X_train, y_train):
    rf = RandomForestRegressor(n_estimators=500, verbose=1)
    rf.fit_transform(X_train, y_train)
    return rf

Also, if there is a better way to make my model accessible via API, that would also be good to know.

Update: I reduced it to 100 trees without losing much predictive power, and so now the saved size is 1.3 GB -- much more manageable, but still not great.


texts: Using LSTM to forecast vehicle position - multivariate time series - Matlab

texts:
I am trying to train an LSTM model on Matlab to forecast the position of a vehicle when driving around a roundabout. My main concern right now is that my dataset consists of 4 features (X position, Y position and 2 other) and I referred to Are RNN or LSTM appropriate Neural Networks approaches for multivariate time-series regression?

and understood the method of sliding for multistep predictions, although intuitively this seems like the right way to do it, my model generates huge errors for some reason - prediction of tn is fairly alright but tn+1 onwards there is an issue similar to this.

Additionally, I have about 200 samples in my dataset ( i.e driving path of 200 different vehicles in the roundabout) how do I go about training a model for all these time-series patterns?

As of now, I am trying to make a model to work on 1 time-series sample and later build on it.

Any suggestion and proposals are appreciated!


texts: Get Decision Tree Prediction With Random Forest

texts:
If I give random forest parameters as

RandomForestClassifier(n_estimators=10,bootstrap=False,max_features=None,random_state=2019)

Should it be creating 10 same decision trees?
But it is not.
I am asking the random forest to

Sample without replacement (bootstrap=False) and each tree have same number of sample (ie the total data )(verified using plot)
Select all features in all trees.

But model.estimators_[2] and model.estimators_[5] are different


texts:
Last few months, I had been exposed to Bayesian Inference in ML course
With further investigation, I come to place where there is MCMC technique to simulate the posterior distribution.
It seems interesting. However, I am not sure if it is really useful in the industry?
Does anyone has experience with Bayesian Inference in practical?
Take customer_lifetime_value as X for example
Basically, my main question is how bayesian inference can be more useful than I just plotting frequency and cumulative frequency of historical X
Because with frequency , i can estimate the mean X,
With cumulative frequency, i can estimate prob[X>x]
What is the advantage of trying to do bayesian inference


texts: Differences between pseudocolor plot and heatmap?

texts: I found some materials using the word `pseudocolor plot' and I think that it's just a heatmap. Is there any difference between them?


texts: Should bag of words in training set include test set data when doing text classification?

texts:
I'm doing text classification to identify 'attacks' from wikipedia comments using a simple bag of words model and a linear SVM classifier. Because of class imbalance I'm using F1 score as my error measure. I'm wondering if the tokens I have in the training data should also include words that exist only in the test data, or does it not matter? I was under the impression that it shouldn't matter, since the counts for these features would be zero anyway in the training set. That should make them irrelevant to the model when training. Apparently that's what some people on SO were saying as well (it didn't find any definitive answer though).
In order to test this I decided to train my model both ways and see the difference: once with only features in the training data, and another with features that included test data. N folds for CV was set to 10. I got a very similar CV error for both of them, but when I generated predictions for my test data my F1 score was 0.06 higher for the model which included features from test data - 0.64 vs 0.58. Because this is a Kaggle assignment I cannot see the true labels for the test set. I'm inclined to believe that such a big difference can't simply be random. It seems like including all the features from test data did improve my model, but how could this be? Can anyone give me an explanation?


texts:
Referring to the LightFM model from paper Metadata Embeddings for User and Item Cold-start Recommendations, the  model tries to learn $d$-dimensional user and item feature embeddings $e_f^U$ and $e_f^I$ for each feature $f$ ($U$ is the set of users, $I$ is the set of items).

The latent representation of user $u$ is given by the sum of its features' latent vectors:
$$
\mathbf{q_u} = \sum_{j\in{f_u}}\mathbf{e_j^U}
$$

The same holds for item $i$:
$$
\mathbf{p_i} = \sum_{j\in{f_i}}\mathbf{e_j^I}
$$

Does it really make sense to sum the latent embeddings to represent a set of features (user or item)?


texts:
I have an assignment, i have to collect movie reviews and classify them with poz neg notr tags, and turn them info arff files. I did all of this stuff with regex etc.

But for experiment i have to choose ten classify algorithm. I did some research but couldn't figure out which algortihm should i use.

Can you please help?
Thanks.


texts: Please list some well tested api's for arima model

texts: I am looking for a good python api for timeseries models such as ARIMA. Please list some well tested apis and few more advance models possible for financial time-series analysis.


texts: How to handle data collecting bias in machine model training

texts:
In many ML problems we collect data and train models using the collected data. Using recommendation as an example, data collected could be biased for various reasons:

presentation bias. For example, on web one may have pagination; for mobile one may need to scroll down to see items beyond a few.
bias from existing recommendation system: usually data is collected on a running production recommendation system, this introduces bias. For example, due to personalization, a male consumer may have little chance to see female products, etc.

What are approaches to handle this when training a model?


texts: Demonstrating and visualizing cluster centroid updates by using the first four iterations in Python?

texts:
I managed to create a dataset with 6 clusters and visualize it with the code below, now I would like to visualize demonstration of update of the cluster centroids in KMeans algorithm. This demonstration should include first four iterations by generating 2×2-axis figure
Here is my code:
# import statements
from sklearn.datasets import make_blobs
import numpy as np
import matplotlib.pyplot as plt
# create blobs
data = make_blobs(n_samples=200, n_features=6, centers=6, cluster_std=1.6, random_state=50)
# create np array for data points
points = data[0]
# create scatter plot
plt.scatter(data[0][:,0], data[0][:,1], c=data[1], cmap='jet',marker="+",label="Original Data")
plt.xlim(-15,15)
plt.ylim(-15,15)
plt.show()

How can I implement this algorithm? It has been asked before using R but, I would like to do it in python. Can you help me visualize the first 4 iterations?
Output should be like:


texts: forecast monthly shipment (time series) for 300 products individually in R

texts:
I can build ARIMA model with regressor to forecast monthly shipment for one product. but I have 300 products and each of them needs a monthly forecast.

my question is, instead of building 300 models, is there another approach in R to deal with the goal I am trying to accomplish?

thank you!


texts:
I have a model like this with multiple outputs and i want to change it's output names

class MyModel(Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.layer_one = Dense(1, name='output_name_one')
        self.layer_two = Dense(1, name='output_name_two')
    def call(self, inputs):
        output_name_one = self.layer_one(inputs)
        output_name_two = self.layer_two(inputs)
        return output_name_one, output_name_two

keras automatically set output names to output_1, output_2, ... how can i change the output names to my desired names?


texts: How does for loops and elif work with python pandas?

texts:
the following code:
df ['weight'] = 1
for feature in df.fclass:
    if feature == 'bus_stop':
        df.weight = 150
    elif feature == 'taxi':
        df.weight = 100
    else:
        df.weight = 50
df.head()

results in the following dataframe:

I expected that the value of the 'weight' column to be 100 in the second column because df.fclass == taxi. Is there something wrong with the for loop or with my 'elif' request?


texts: Can we use Cauchy-Schwarz Divergence loss for binary classification tasks?

texts: Cauchy-Schwarz Divergence loss. From here


texts: Custom objective function in xgboost for Regression

texts:
I am working on a regression problem, where I want to modify the loss function in xgboost library such that my predictions should never be lesser than the actual value. I have written this code:

def custom_loss(preds, dtrain):
    labels = dtrain.get_label()
    df = preds - labels
    df = pd.DataFrame(df, columns=['val'])
    df['valg'] = df['val'].apply(lambda x: 10*abs(x) if x<0 else x)
    grad = df['valg'].as_matrix()
    return preds-labels, grad

This essentially means that I want to penalize those predictions which are lesser than my actual value more. However, this isn't working and there is no improvement in my predictions. Can anyone help me figure out where I am going wrong? Thanks.

Whole Python script:

params = {"booster" : "gbtree",
  "eta": 0.20,
  "max_depth": 4,
  "subsample": 0.75,
  "colsample_bytree": 0.65,
  "silent": 1,
  "eval_metric": "rmse",
  }
num_round = 400

def custom_loss(preds, dtrain):
    labels = dtrain.get_label()
    df = preds - labels
    df = pd.DataFrame(df, columns=['val'])
    df['valg'] = df['val'].apply(lambda x: 5*abs(x) if x<0 else x)
    grad = df['valg'].as_matrix()
    return preds-labels, grad

dtrain = xgb.DMatrix(X_train.drop('price_act', axis=1),
label=X_train['price_act'])
dtest = xgb.DMatrix(X_test.drop('price_act',axis=1),
label=X_test['price_act'])

watchlist = [(dtrain,'train'), (dtest,'eval')]

bst = xgb.train(params, dtrain, num_round, watchlist, custom_loss)


texts: How do you create an optimized walk list given longitude and latitude coordinates?

texts: I am working on a political campaign where dozens of volunteers will be conducting door-knocking promotions over the next few weeks. Given a list with names, addresses and long/lat coordinates, what algorithms can be used to create an optimized walk list.


texts: How to group a unique column and find the mean value in a separate column in R

texts:
My table of data looks like this image (but with many more lines):

I want to be able to find the mean age of each region, so the mean age for region SSC20790 and so on. I believe I can use the dplyr package?


texts: Extract person's academic degree from text

texts:
I have a corpus of free form text (emails) and am trying to extract the highest degree (eg. High School, Bachelor, Master's, phD) from each of them. Is stemming the way to go? Or lemmatization? Note that the degree may not be mentioned at all.

Some example emails that I'll need to go through:

  Hi. My name is XXXXX. I have bachelor's in Math from UPenn and a Master's of Education from Stony Brook. Would love to work with XXXX!

  Dear XXXX, my name is XXXX and am interested in your company. Please email me for CV.


texts: Evaluating Logistic Regression Model in Tensorflow

texts:
Following this tutorial, I have a doubt about the evaluation part in:

# test the model
n_batches = int(mnist.test.num_examples/batch_size)
total_correct_preds = 0
for i in range(n_batches):
    X_batch, Y_batch = mnist.test.next_batch(batch_size)
    _, loss_batch, logits_batch = sess.run([optimizer, loss, logits], feed_dict={X: X_batch, Y:Y_batch})
    preds = tf.nn.softmax(logits_batch)
    correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(Y_batch, 1))
    accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) # need numpy.count_nonzero(boolarr) :(
    total_correct_preds += sess.run(accuracy)

print 'Accuracy {0}'.format(total_correct_preds/mnist.test.num_examples)

Note that this is done on the test set, so the goal is purely to obtain the accuracy using a previously trained model. However isn't calling the line:

_, loss_batch, logits_batch = sess.run([optimizer, loss, logits], feed_dict={X: X_batch, Y:Y_batch})

equivalent to re-optimizing the model using the test data (and labels)?
Shouldn't we avoid re-running optimizer and loss and just compute the predictions?


texts: How to customize word division in CountVectorizer?

texts:
>>> from sklearn.feature_extraction.text import CountVectorizer
>>> import numpy
>>> import pandas

>>> vectorizer = CountVectorizer()
>>> corpus1 = ['abc-@@-123','cde-@@-true','jhg-@@-hud']
>>> xtrain = vectorizer.fit_transform(corpus1)
>>> xtrain
<3x6 sparse matrix of type '<class 'numpy.int64'>'
    with 6 stored elements in Compressed Sparse Row format>

>>> xtraindf = pd.DataFrame(xtrain.toarray())
>>> xtraindf.columns = vectorizer.get_feature_names()
>>> xtraindf.columns
Index(['123', 'abc', 'cde', 'hud', 'jhg', 'true'], dtype='object')

I see that the special characters(-@@-) are omitted and "abc" and "123" are considered seperately. But, I want "abc-@@-123" to be treated as a single word. Is it possible to achieve? If yes, how?

Any help would be much appreciated.


texts: Compare Coefficients of Different Regression Models

texts:
in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be able to compare the coefficients of each regression model between each other. Example: I have

Lasso
Ridge
ElasticNet
AdaBoostRegressor
GradientBoostRegressor

I am aware that I could get the coefficients of Lasso, Ridge, and ElasticNet from model.coef_ and model.intercept_ from sklearn. However, AdaBoostRegressor does not have this, but rather, it has weights assigned to each of the estimators. Similarly, GradientBoostingRegressor of sklearn does not even have weights at all. How am I to compare the coefficients between each of the regressive models I am using??


texts:
I would like to find the two furthest pages of a big websites (+5millions pages).

(It's the degree of separation, the most links between the two pages)


texts: Best Way to combine multiple datasets into one model

texts: I want to make a multilabel image classification model that can detect many different labels. For each label, I can get at least 5000 positive examples and 5000 negative examples. However, my question is how can I use data in this format to train a multilabel image classifier. Part of the challenge is, for example, I can download 10,000 images of a hand and know that they are positive examples but then if I want to detect a shoe as well, I don’t know how many of those hand photos might have also had a shoe in them. I’m trying to make a model this way because I will have a fairly high amount of labels and need to be continuously adding new labels. What is the best way to go about doing this?


texts: The meaning of the difference of two entropy values

texts:
I want to understanding the meaning of the difference of two information entropy values.

I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, we may have the sets:

$X_{A} = \{x_1,x_2,\cdots,x_{n_A}\}$, and
$X_{B} = \{x_1,x_2,\cdots,x_{n_B}\}$ that represent the number of hours the users of $A$ and $B$ spent on the websites $A$ and $B$, respectively.

Now, we can calculate the Cumulative distribution function
 (CDF) probability values, described here, as follows:
For a real-valued random variable X, the CDF is defined as:

$$F_A(x) = P(X\leq x),\ \forall x \in X_A$$

Similarly, we find $F_B(x)$.

Then we calculate the entropy values $E_A$ and $E_B$ for the probabilities  $F_A(x)$ and $F_B(x)$ obtained above (as described here):

$$E_A = - \sum\limits_{i=1}^{n_A}P(x_i)log(P(x_i))$$

Similarly, we find $E_B$.

Now my question is, what do $E_A$ and $E_B$ mean in this scenario? and what their difference also means?


texts:
I am trying to remove stop words before performing topic modeling. I noticed that some negation words (not, nor, never, none etc..) are usually considered to be stop words. For example, NLTK, spacy and sklearn include "not" on their stop word lists. However, if we remove "not" from these sentences below they lose the significant meaning and that would not be accurate for topic modeling or sentiment analysis.

1). StackOverflow is helpful      => StackOverflow helpful
2). StackOverflow is not helpful  => StackOverflow helpful

Can anyone please explain why these negation words are typically considered to be stop words?


texts: tuning a convolution neural net, sample size

texts:
I keep reading that convolution neural net (CNN) performs best with lots and lots (100k+) of data.  Is there any rule of thumb, or lower limit for data size during the grid search phase?

For example, if I run a CNN with 100 data points, vary just one parameter (say add an extra layer, or increase a filter size), and get better results, can I reasonably expect better results with those parameters during the actual training phase?


texts: Time Warping - Procrustes Canonical Time Warping Alignment stuck on first entry

texts:
Context: Working on Temporal Alignment for sign-language features.

I'm comparing three methods for time-warping (Uniform Time Warping - UTW, Procrustes Dynamical Time Warping - PDTW, Procrustes Canonical Time Warping - PCTW) on some and get results, which are hard to explain for myself. I expected the PDTW and PDTW to be at least somewhat comparable, but in the results are vastly different. This is mainly, because the PCTW gets stuck on the first entry:

Now I looked into it in some more detail, occasionaly it also might get stuck on another index later on, but only rarely, and comparably a lot shorter. I can upload graphs of that as well if needed.
I use the MATLAB implementation by Zhou and De la Torre based on this paper.

I'm really grateful for any idea how to interprete and possibly fix this.


texts: How to do prediction on survival data, using Random Forest

texts: I should make prediction on survival data, using the random Forest method. My question is: should I follow the same approach as in logistic regression? taking into account only the status variable or whether I should take into account the delay to the event?  Are there any specific R functions for survival analysis other than randomForest? Or could I use this function for survival analysis as well? I've seen a function called ranger() that seems to do random forest on survival data, but I haven't understood much of it.


texts: How important is the balance between the components of a loss value?

texts: I'm training a neural network for predicting the location of a single object, so the prediction consists of 4 values: x, y, width and height. Without weight decay, the loss starts at around 0.3. If I add weight decay with a factor of 0.004, the loss jumps to around 22. Does this mean the network will give more importance to minimizing the weights over finding good coordinates? How important is the balance between the components of a loss function?


texts: Sum of squares for matrix valued data over $\mathbb{R}$ and $\mathbb{C}$

texts:
Let us assume we have $k \times k$ matrix valued data and assume this is organized (possibly as time series):
$$ M_1, M_2, \ldots, M_n $$
Now, assume we are interested in writing down an error function that mimics sums of squares. This can naively be written as
$$ \sum_{i=1}^n (M_i - \hat M_i)^2 $$
where $\hat M_i$ is the $i$-th estimation. The question is, what is actually the proper way to write this function explicitly? For vectors, the Euclidean norm is "naturally" picked. What about this case?
One option is to multiply out these matrices and treat each of the resulting matrix's elements on its own. For example the element at position 11 would have its own "error function" that looks like:
$$
\sum_i (a_{11}^2 +a_{12}a_{21})
$$
and similarly for the other three elements. Here $M-\hat M \equiv A = (a)_{ij}$. Does this even make sense?
Furthermore, how to treat the same example having complex valued matrices?


texts: What is the best to identify the proper hierarchy of this data?

texts:
So I worked on a hierarchical clustering algorithm to be able to determine which items are most similar, and what attributes are most important. I have two tables:

Table 1: contains a bunch of item codes, and it's attribute (brand, flavor, sales, and so on). It looks something like:

Item_code | Brand | Flavor   | Caloric_content | ... | sales
006891313 | Coke  | Original | 0               | ....
002349823 | Fanta | Orange   | 200             | ...

The other table that I have is what i run my clustering algorithm on. It's an NxN matrix, (where N is the number of distinct item_codes in the previous table) An entry [i,j] in the matrix, corresponds to the number of times j was bought after i was purchased on a previous trip. So more clearly, in the matrix below, what the number 1223 means is that, 1223 times after item 003428734  was purchase on an initial trip to the store, item 003428734 was purchased on the next trip.

          |003428734 |009849328 | 09840202 |....
003428734 |  1223    |    13    |    0     |
009849328 |    12    |   945    |    34    | ....
   .
   .
   .

I apply a hierchical clustering on that matrix, using Ward 2, squared euclidean distance and standard Z scores. The final output is a dendrogram, with all the item_codes on the branches of the dendrogram.

This is where the tedious process is. The ultimate goal, is to have a hierarchy for our products, and see which attribute (brand, flavor, size...) falls where on the dendrogram. The only way i can think of doing it, is organizing the item codes in the first in the same order that they are in the dendrogram, put a picture of the dendrogram side by side, and eyeball which attribute clusters the most. You'll see in the link below, it looks like brand A and B are clustering together, so I would say that products cluster by brand first. I would then take a closer look at the smaller cluster within the dendrogram and try to identify where the other attributes fit in the hierarchy (the picture is just a snippet of the dendrogram, it is in reality much, much bigger)

This eye balling process is very tedious and annoying. Is there a way to run a similar cluster, which shows a hierarchy within the products, as well as indicate which attribute fit where in that hierarchy? Or is this even the best way to approach this?

I should mention that I'm very new to these clustering algorithm, so apologies if this is a dumb question


texts: I have X numpy array as my features and y numpy array as my target. I split both of it into train and test data. From many QnA i have read they only say to preprocess both train and test separately. I assume i only do it to my feature (X) train and test data and not the target (y). Do we also preprocess the target?


texts: Why maximize ELBO in the variational autoencoder?

texts:
For a variational autoencoder, we have that:

$$\mathcal{L}(x,\theta,\phi) := \mathbb{E}_{z \sim q_\phi(z|x)}[\log p_{\theta}(x|z)] -KL[q_{\phi}(z|x) ||p(z)] $$

This is called the variational lower bound or evidence lower bound (ELBO). But I think what we're actually trying to maximize is the log-likelihood of our data:

$$\log p_{\theta}(x) = \mathcal{L}(x,\theta,\phi) + KL[q_{\phi}(z|x)||p_{\theta}(z|x)]$$

There are a few things I'm unsure about, in increasing order of difficulty.

For the actual loss function of a VAE, we use $-\mathcal{L}$, more or less. Of course, it's expensive to actually calculate the expectation, which is why we use a single $z$ sample each time, yes?
We are told to treat $p(z)$ as being $\mathcal{N}(0,1)$, but I don't see what causes it to become normal. We're just told to plug in $\mathcal{N}(0,1)$ for $p(z)$ when calculating the loss, and all this seems to do is ensure that $q_\phi(z|x)$ gets closer to it. We know nothing about $p(z)$, right (besides the fact that if all $p_{\theta}(z|x)$ are close to some distribution, then so is $p_{\theta}(z)$)?
How should we think about that KL divergence in the second formula? It's usually pointed out that for fixed $p_{\theta}$, maximizing $\mathcal{L}$ is equivalent to minimizing the KL div. But $p_{\theta}$ is not fixed; it's being learned. We could also improve $\log p_{\theta}(x)$ (our ultimate goal) by making that divergence worse, couldn't we? What justifies optimizing $\mathcal{L}$ alone (other than tractability)?

Basically, I'm confused by the common explanation that what we really want is to minimize that second KL divergence, and also that the best way to do that is to maximize the ELBO.


texts:
I have a dataset of patient records but they don't have labels.

I would like to label them and would like to know what are the different approaches available that I can consider to label them.

For ex:  I have given two approaches below but is there any other approaches to it?

1) Manual labeling

2) Rule based

3) ..

4) ..

As you all are experienced in data science and AI

Can you please share some approaches that you know and resources that I can refer to learn about them?


texts: Segmentation of time series into one-second non-overlapping windows

texts: I need to split the time series dataset (accelerometer values (timestamp, X, Y, Z)) into segments in the form of a one-second window that does not overlap. I am trying to find an example of its implementation, but found nothing. I'm a beginner and don't know how to do this.


texts:
I have a data as below:

Date                                 Quantity   Rate      Value
01-Jan-18   0.5HP Motor Xcite        9 nos      2277.82   20500.39
02-Jan-18   0.5HP Pump Booster       1 nos      11648.30  11648.30
03-Jan-18   0.5 HP Trashmasher       1 nos      3390.70   3390.70

I want to predict the future value of the quantity/rate/value based on particular date for the given particular.  Which model can be used to predict the future value for the given particular for a given date in the future? (I am a R developer and just started and I don't know Python)


texts: What is the best way to normalize a set of datasets

texts: I have a data set that contains the same Time series "Sensor readings" for different days and I want to make a deep learning model to predict these values. What I did was I splatted the data into Time series data according to the day, then I normalized it separately (min-max) (the readings have different ranges, for example, the max value for the first day is 100 but the max for second is 48) but I'm really confused now do I need to normalize it using the max/min of the all days or what I did was right?


texts: Does dataset training and test size affect algorithm?

texts:
I am dealing with a dataset that has a total of 81 records. Out of them I divided it to 54 records for the training and the rest for testing. I noticed a few things:

Accuracy is low for given test data set (~ 30%).
Accuracy remains same no matter whether I use SVM or Naive Bayes.
If I test against 2-3 values accuracy shoots up to 75%.

Is this normal or I am doing something wrong?


texts: How to implement Semantic Search in R or Python

texts: I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-related skills despite of missing "machine learning" keyword. How do we search the data through its meaning and related keywords I wonder? I have checked many algorithms also Like LSA, LDA, LSI but cannot find a resource which gives the implementation of the above.


texts: How to use df.groupby() to select and sum specific columns w/o pandas trimming total number of columns

texts:
I got Column1, Column2, Column3, Column4, Column5, Column6
I'd like to group Column1 and get the row sum of Column3,4 and 5
When I apply groupby() and get this that is correct but it's leaving out Column6:
    df = df.groupby(['Column1'])[['Column3', 'Column4', 'Column5']].sum

I tried with this but it doesn't group according to Column1 and it doesn't sum anything, but I get all my columns:
    df.sort_values(['Column1']).groupby(['Column3', 'Column4', 'Column5']).sum()

How can I use groupby() correctly in this case?
Thank you!
I add my code:
    df = df.drop(['Position', 'Swap', 'S / L', 'T / P'], axis=1)
    df = df.groupby(['Symbol']).agg({'Profit': ['sum'], 'Volume': ['sum'], 'Commission': ['sum'], 'Time': pd.Series.mode})
    df['Comm. ratio'] = (df['Commission'] / df['Profit'])
    df['Comm. ratio'] = df['Comm. ratio'].round(2)
    df['Net profit'] = (df['Profit'] + df['Commission'])
    df.loc['Total'] = pd.Series([df['Commission'].sum(),df['Profit'].sum(),df['Net profit'].sum()], index = ['Commission','Profit','Net profit'])

The output is:

As you can see it adds "sum" and "mode" rows that I'd like not to have.
Moreover, it ignores the df.loc['Total'] code and it leaves the Total row empty.


texts: binning high cardinality categorical features

texts:
one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently occurring values (with x depending on the frequency distribution), and then create a binary flag feature like 'is_in_top_x_us_cities'. Or creating one-hot features for each of the top x cities.

Can someone explain the relative disadvantages to this approach as opposed to using something like Weight of Evidence binning?


texts: Is there an upper bound for k in nearest neighbors-based methods?

texts:
When applying a nearest neighbors-based method to a data of, for instance, 2000 points, what is the largest number of neighbors that can be considered ?
I am using a nearest neighbors method in an unsupervised fashion for anomaly detection.


texts: How to migrate application which has backend running using DLL methods and front end in JAVA to Apache spark?

texts:
I have one Java Client which Reads 4 to 5 GBs of data and uses DLL file in order to process it and give O/P back to the JAVA client.
Can anyone tell me if this can be implemented using Apache Spark or not ?
And if yes can you back you Answer by some examples or documentation Links ?

EDIT

I have knowledge about JNI and how to do this in windows environement.
What I'm actually looking for is How do you implement such thing in Apache Spark.

As all methods and processing is done via DLL files and JAVA Client is just having represntational logic so any documentation is helpful.

I'm currently using windows development env. but gonna use Ubuntu 14.04 for Apache Spark implemetation.

EDIT after Hold

This question is really intending to ask where to find such documentation or help with which we can migrate such application smoothly.


texts: I have around 1600 images extracted from videos shot at night time. I am labeling each image and trying to be as accurate as I can in assigning bounding boxes. I am labeling vehicles and traffic light/traffic signs. This is very time-consuming, I am wondering if someone have experience or done this before and can advise me on some automated methods of labeling night time images. The objects of interests are usually appear to be quite small. An example labelled image is given below


texts: What's a reasonable distribution to model views over time of... this question?

texts:
Sorry for the self-referential title ;)

I'd be curious to know what's a curve that could be used to model - in the scale of (say) weeks* - the "engagement lifecycle" of a single news page of an online newspaper in terms of webpage views. The same model could be applied for Stack Exchange questions views too!

What I would intuitively model is the initial spike of visits due to the fact that a new, fresh page is very visible as it's linked in the homepage (triggering also social media reshares) and then the long-term visit behavior e.g. due to that page being reached from links from other webpages, or via search engines.

One possible way would then be the sum of two Poisson processes (the first switching off after some typical characteristic time).

Another possibility would be using something like exponential decay to capture a decay over time of interest (the "freshness" of the content fades, and so does public interest).

Is anyone aware of better alternatives or literature on this?

*I wouldn't model day-night nor weekly seasonality.


texts: In multiple linear regression why is it best to use an $F$-statistic when evaluating predictors?

texts:
I am currently going through Hastie and Tibshirani's 'Introduction to Statistical Learning' textbook and I have come across something I don't understand on page 77. I have two questions.

The author states that if we had 100 variable predictors (with respective coefficients $\beta_i$) and supposed that the null hypothesis, $$H_0 : \beta_1 = ... = \beta_{100} = 0$$ were true, then roughly $5\%$ of the $p$-values would fall below $0.05$ by chance, and therefore we might wrongly conclude that certain predictors are related to the response. Why would this happen by chance? Is this simply a mathematical truth?

In addition, the author then goes on to state that the $F$-statistic is a better measure because "if $H_0$ were true, then there is only a $5\%$ chance that the $F$-statistic would result in a $p$-value below $0.05$". I don't understand the difference - could somebody explain a bit more clearly?


texts: How to normalize a boolean feature for neural nets?

texts: I have a feature that is boolean and I would like to feed it to a neural net as one of the inputs. I think in theory the best is to encode as false->0 and true->1 because 0 as an input will deactivate weights of a neuron. Is this correct?


texts:
When I write a network, do I have to write the whole forward property in nn.Module.forward()? I mean if I do some operations outside the net, does grad correctly flow?

For example, I have two networks, in which the output of one is the input of the other(net1 -> midresults -> net2).

If I do some operations on midresults(net1 -> midresults -> operations -> net2), can (net1+net2) be trained end to end?


texts: What are some good books on Machine Learning and AI like Krugman, Wells and Graddy's "Essentials of Economics"

texts: I am a Logistics student. I like the book "Essentials of Economics" by Krugman, Wells and Graddy in that it is concise, easygoing and not only a beginners book (it gradually approaches advanced subjects thus paving the way for further rigorous Economics course) so any one interested in Economics can learn it even if he/she never studied the subject before. Also, I am very interested in AI and Machine Learning and acknowledge their importance in this our postmodern era and I am self learning Real Analysis and web site development. What are some good introductory books on Machine Learning and AI like Krugman, Wells and Graddy's "Essentials of Economics"?


texts: How to implement custom loss function that has more parameters with XGBClassifier in scikit-learn?

texts:
I have following problem with implementing custom loss function with scikit-learn:

I would like to implement Focal Loss as my objective function in XGBClassifier. However, I dont know how to pass additional arguments as a parameter(objective parameter):

 def focal_loss(y_pred, y_true, alpha=0.25, gamma=1):
  a,g = alpha, gamma
  def fl(x,t):
    p = 1/(1+np.exp(-x))
    return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )
  partial_fl = lambda x: fl(x, y_true)
  grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)
  hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)
  return grad, hess

xgb = xgb.XGBClassifier(objective=focal_loss)

What should I do in following situation? Is there maybe ready version of Focal Loss ready to use? Thanks in advance.


texts: How to approach clustering of time-series with a single variable

texts:
Let me preface this by saying that I'm a complete beginner to R and data science in general, so my apologies if this is a rather trivial question. I do have a rough idea of what I would like to achieve though, and how the end result should look like. I just don't know which algorithm is best suited for dealing with this kind of data and how to go about implementing it.

I would like to run a clustering algorithm on a data frame with 13369 distinct units, each having just a single variable that is tracked over 15 years. So my data looks something like this:

id  y1  y2  y3  y4  y5  y6  y7  y8  y9  y10 y11 y12 y13 y14 y15

01  0   6   0   3   0   0   0   0   0   0   0   1   1   0   0
02  1   4   1   6   4   3   7   6   3   8   11  10  9   10  10
03  0   0   2   0   5   0   1   0   3   0   0   0   0   0   0
...
...
...
13369 4  9  0   12  5   1   1   0   6   1   2   7   0   0   3

To establish some context for this task, this is the data for authors and the number of publications they have published in each year, spanning over 15 years.

I would like to end up with a dendrogram that depicts various clusters that authors fall into, (I'm guessing) depending on how prolific they were in a certain time period (beginning vs. later years, for example).

I hope I explained everything clearly.
Thank you for taking your time and assisting a student in need.


texts: High-dimensional data: What are useful techniques to know?

texts:
Due to various curses of dimensionality, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/heuristics that help deal with high-dimensional data effectively? For example,

Do certain statistical/modeling methods perform well on high-dimensional datasets?
Can we improve the performance of our predictive models on high-dimensional data by using certain (that define alternative notions of distance) or kernels (that define alternative notions of dot product)?
What are the most useful techniques of dimensionality reduction for high-dimensional data?


texts: Is it necessary to sort time series data on datetime stamps?

texts:
Say I have a time-series data frame for a bank that records number of cases filed for 16 districts as below (cases can be for many purposes - loans, credit cards, real estate, etc.):

application_date    branch_id    district    case_count
2017-04-01              1        A           40.0
2017-04-03              1        A           5.0
2017-04-04              1        A           4.0
2017-04-05              1        A           113.0
2017-04-07              1        A           76.0

But if I sort the above data by application_date, like so:

application_date    branch_id    district    case_count
2017-04-01              1         A           40.0
2017-04-01              62        B           5.0
2017-04-01              63        C           4.0
2017-04-01              64        D           113.0
2017-04-01              65        E           76.0

Considering that branch_id and district are categorical variables, does sorting affect performance of standard ML models like random Forests and gradient boosted models as well as their predictive accuracy?


texts: An Unsupervised learning method suitable for large categorical data sets

texts:
I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had more than 90 percent missing values.
This data set tries to detect anomalies through unsupervised learning. I'm currently using Autoencoder to access it, but I wondered if this would work. Also, because the purpose is to detect whether data is abnormal when data comes in in real time, clustering techniques such as dbscan are limited.
If you want to apply this categorical and many missing values to unsupervised learning, I am curious about how to organize the data and whether there is an appropriate unsupervised learning method.
Since the dataset is the data used in the project, we don't currently have it.


texts: Why is word prediction an obsession in Natural Language Processing?

texts:
I have heard how great BERT is at masked word prediction, i.e. predicting a missing word from a sentence.

In a Medium post about BERT, it says:

  The basic task of a language model is to predict words in a blank, or it predicts the probability that a word will occur in that particular context. Let’s take another example:

  “FC Barcelona is a _____ club”

Indeed, I recently heard about SpanBERT, which is "designed to better represent and predict spans of text".

What I do not understand is: why?

I cannot think of any common reason that a human would need to do this task, let alone why it would need to be automated.
This does not even seem to be a task where it is particularly easy to evaluate the success of a model. For example,

  My ___ is cold

This could reasonably be a number of possible words. How can BERT be expected to get this right, and how can humans or another algorithm be expected to evaluate whether "soup" is a better answer than "coffee"?

Clearly there are a lot of smart people who think that this is important, so I accept my lack of understanding is likely based on my own ignorance. Is it that this task itself is not important, but it's a proxy for ability at other tasks?

What am I missing?


texts: Is there any way to explicitly measure the complexity of a Machine Learning Model in Python

texts:
I'm interested in model debuggin and one of the points that it recommends is to compare your model with a "less complex" one in order to see if the performance is substantially better on the most complex model.
It raises my questions:
Suppose you have a Ensemble model and a Linear model for a classification task
"It seems natural to think that the ensemble model is more complex than the linear model"

But, what would it be a model agnostic way to numerically measure the model's complexity in order to be able to compare two or more models in such terms?

Is there any python implementation that can help with such a task?


texts: In linear regression, is there anything I can do if the coefficient for one of the features is unrealistic/inappropriate?

texts:
I'm building a simple linear regression model that predicts Home Price using Square Footage, Number of Bed(s), and Number of Bathroom(s).

After creating the model, I noticed that the coefficients for Square Footage and Number of Bed(s) were positive, which makes sense since Home Prices increases as Square Footage/Number of Bed(s) increases. However, the coefficient for Number of Bathroom(s) was negative, which makes no sense since Home Price does not decrease as Number of Bathroom(s) increases!

My question is this: is the negative coefficient something I can "fix" by incorporating more data into my model, or is this just a quirk in linear regression?


texts: Storing engineered features in a database

texts:
I have some data in raw csv files which I would like to store in a MySQL database.  The problem is there are constant feature engineering done on this dataset so coming up with one schema to fit all the needs is not possible.  The approach I thought of was to have one main table where the original data is held, and for each new feature that is created, a new table is created.  Then, the user can join the original table with other tables which includes the features they want and use it for their own purposes.

With the above approach, I'm worried about having too many joins when a user needs numerous features.  Please advise on alternative approaches to this problem.

Thanks in advance!


texts: python - Identify variable in similar sentences

texts:
I'm looking to solve the following problem: I have a list of similar sentences as my dataset, and I want to be able to type a new sentence, which is also similar to the sentences in my dataset and then highlight the parts of the sentence which are varying. For example,

This is the set I have:

I want to eat an apple
I want to eat an ice cream
I want to drink coffee
I want to eat a pizza
I want to puke ginger

These sentences are similar but they have some varying parts. The two main variables in these sentences are the action (eat, drink, puke) and the food (apple, ice cream, coffee, pizza, ginger).

Now if I type in a new sentence say I want to drink cola I want my model to identify drink and cola as the variables in this sentence.

At this point, I don't really care about the semantics that much. I will work on that later. Even if the sentence is I want to hug a table, all I want to do is identify hug and table as variables.


texts: Identifying repeating sequences of data in byte array

texts:
Given a sample of hexadecimal data, I would like to identify UNKNOWN sequences of bytes that are repeated throughout the sample. (Not searching for a known string or value) I am attempting to reverse engineer a network protocol, and I am working on determining data structures within the packet. As an example of what I'm trying to do (albeit on a smaller scale):

(af:b6:ea:3d:83:02:00:00):{21:03:00:00}:[b3:49:96:23:01]

{21:03:00:00}:(af:b6:ea:3d:83:02:00:00):01:42:00:00:00:00:01:57

And

(38:64:88:6e:83:02:00:00):{26:03:00:00}:[b3:49:96:23:01]

{26:03:00:00}:(38:64:88:6e:83:02:00:00):01:42:00:00:00:00:00:01

Obviously, these are easy to spot by eye, but patterns that are hundreds of chars into the data are not. I'm not expecting a magic bullet for the solution, just a nudge in the right direction, or even better, a premade tool.

I'm currently needing this for a C# project, but I am open to any and all tools.


texts: Like Word2vec is not a single algorithm but combination of two, namely, CBOW and Skip-Gram model; is Doc2Vec also a combination of any such algorithms? Or is it an algorithm in itself?


texts: What’s the best way to save many pandas dataframes together?

texts:
I’m looking for a way to save house prices data by city, for example a pandas panel with one dataframe per city. But I need the dataframes to be independent, meaning that if one dataframe is corrupted, the others are untouched.
I tried using pickle and csv, but once a line was corrupted I lost the whole file.


texts: predict_proba() on a continuos target made binary

texts:
I am building a Newtons gravitation equation model. It receives two mass and a radius and outputs what it generates from the formula.
$F = G \frac{m1 · m2}{r^2}$
I want to transform it now to a binary classification model, for that I select a threshold. If the force is above then 1 if below 0.
The question
How can I create a predict_proba function given the above problem?


texts: Finding lookalike customers for Digital Media targeting

texts:
I want to develop a model for digital marketing team. We have some set of ideal customers to whom we to send our promotions and have been converted.
Now we want to pick past such cases and want to send promotions to the similar (lookalike) customers for digital marketing.
I know its a broad question but any approach, techniques and suggestions would be appreciated.


texts: One multilabel classifier or one for each type of label?

texts:
Let's say I need to classify addresses with scikit-learn, so if I want my classifier to be able to classify addresses by the street name, and post/zip code, should I do a OneVsRest classifier, or separate them into two different classifiers (for the same training set)?

I have tried both, and it seems like having multiple classifiers might be a better choice, as it feels faster to train multiple smaller classifiers. Is this how it is supposed to be done?


texts: How do you Show a Difference between Two Groups (Clustering)

texts:
I am approaching a data problem. My data set consists of observations of (X,Y) coordinates indicating a position on some grid. There are two groups based on a variable Z. Group A is all the points where Z>10 and Group B is all the points where Z<10.

Would it be a good idea to separately cluster the 2 groups of (X,Y) coordinates, then compare the location of clusters to reach some conclusion about how Z affects the location of the points?

I want to say something statistically about how differently, if at all, the group A and group B points are clustered. So I can say that the difference is because of Z.


texts: What tools are out there to collect participants' browsing and/or search data as part of an experiment?

texts:
I'm running an experiment where I need to collect and analyse participants' browsing and search histories. The design of the experiment is similar to an "instrumented user panel", described here:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.8971&rep=rep1&type=pdf

In the classic case, participants must install some kind of logger on their computers, which collects and sends browsing data to the researcher behind the scenes. Finding such tools is where I get stuck.

I could, of course, just ask my participants to export their browsing histories and send them to me every night, but I was hoping there would be something more automated out there, with potential bells and whistles such as annotations.

Note: Please let me know if this is the wrong Stack Exchange; I must be really struggling with finding the right keywords for this...


texts: My CNN model Accuracy doesn't increase (high loss and low acc)

texts:
Well, I need to do a CNN to classify if a Image is from one or another class. But my model return high losses (6.~8.) and low accuracies (0.50 on max). I tried to include more layers, change my activation functions, and nothing works. My database is 142 .jpg imgs (71 for each class)

This is my code:
OLD CODE

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

    # Get all subdirectories
    FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]
    print(FolderList)

    # Loop over each directory
    for File in FolderList:
        for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
            # Convert the path into a file
            ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (600,700)))
            LabelList.append(classes.index(os.path.splitext(File)[0]))

    return ImageCV, LabelList

model = Sequential()
model.add(Conv2D(64, kernel_size=(3,3), padding="same",activation="relu", input_shape=(700,600,3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128,  kernel_size=(4,4), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='RMSprop', loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)

model.fit(np.array(data), np.array(labels), epochs=10, batch_size=20)

model.save('model.h5')

What can I do to improve my model? I appreciate your help!

UPDATE
I tried to do what Shubham Panchal said but isn't resolve the problem:

THINGS THAT I TRIED
- Reduce Img size
- lr=0.0001
- optimizers: adam, sgd, rmsprop
- put more layers
- put dropout layer
- normalize the data with np.array(data) / 255.0
- Increase the data (1400 total, 700 each class)

My code:

model = Sequential()
model.add(Conv2D(64, kernel_size=(3,3), padding="same",activation="relu", input_shape=(150,150,3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(64, kernel_size=(3,3), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128,  kernel_size=(3,3), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128,  kernel_size=(3,3), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(1, activation='softmax'))
opt = SGD(lr=0.0001, momentum=0.9)
model.compile(optimizer = opt, loss="binary_crossentropy", metrics=['accuracy'])
#model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)

model.fit(np.array(data) / 255.0, np.array(labels), epochs=10, batch_size=16)

My console:

Epoch 1/10
1400/1400 [==============================] - 58s 42ms/step - loss: 7.9712 - acc: 0.5000
Epoch 2/10
1400/1400 [==============================] - 59s 42ms/step - loss: 7.9712 - acc: 0.5000
Epoch 3/10
1400/1400 [==============================] - 59s 42ms/step - loss: 7.9712 - acc: 0.5000
...

Anyone have any ideia what can I do??


texts: Generalization bound (single hypothesis) in "Foundations of Machine Learning"

texts:
I have a question about Corollary $2.2$: Generalization bound--single hypothesis in the book "Foundations of Machine Learning" Mohri et al. $2012$.

Equation $2.17$ seems to only hold when $\hat{R}_S(h)<R(h)$ in equation $2.16$ because of the absolute operator.  Why is this not written in the corollary?  Am I missing something important?

Thank you very much for reading this question.


texts: Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data?

texts: I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, you need high dimensional data. Another lecture, this one on sigmoid in neural nets context (MIT S.191), suggested sigmoid's non-linear behaviour is the reason why a neural net can model a non-linear decision boundary. So, I'm a bit confused. Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data because you can use the sigmoid to produce a linear decision boundary w/o incident?


texts:
Imagine having a large database coming from a super market chain with panel data, whose cross-section variables are the different products sold in each super-market. The frequency is daily, that is you have one value for each product sold in each super-market per day.
Beyond their own history, you have the time-series of other variables e.g. dummy variables that signal the existence of offers.

Your objective is to predict the number of goods sold for each product in each super-market.

Which functions would you use in R to make a week-ahead multivariate forecast?
Which approach would you suggest?


texts: Is there any formula between the number of training inputs, the number of features and the number of epochs that's enough to learn the model?


texts:
I have a data set which has continuous independent variables and a continuous dependent variable. To predict the dependent variable using the independent variables, I've run an ensemble of regression models and tried to compare them against each other. Here are the results for reference:

I can interpret what the R-squared value / Coefficient of determination for each of those models means. However, I can't understand what the Negative Log Likelihood means. Especially, why is it Infinity for Linear Regression and Boosted Decision Tree, and a finite value for a Decision Forest Regression?

Edit:
Data Description: The data that went into these three models is all continuous independent variables and a continuous dependent variable. There are a total of 542 observations and 26 variables.
These 542 variables are split 70 - 30 to get training and testing datasets. Therefore, the training dataset has 379 observations and 26 variables; the testing dataset has 163 observations and 26 variables.
No missing data.

Edit 2 Possible Explanation - (click here): Apparently, Linear Regression and Boosted Trees in Azure ML don't calculate the Negative Log-Likelihood metric - and that could be the reason that NLL is infinity or undefined in both cases.


texts: Regression in Predicting Tenancy Lengths

texts:
I'm currently working on a project involving the prediction of tenancy lengths. I've so far managed to get to a point where I've processed the data and pruned my Random Forest model (via sklearn in Python) to the following accuracy levels (in days):

Train MAE: 131
Train R^2: 0.906
Test MAE: 259 (using cross-validation)
Test R^2: 0.651

While the model is decent for the industry, there's more performance to squeeze out of it. It currently overestimates results and has poor accuracy on the test data imo.

I'd like to further develop a Neural Network approach, as my initial implementation of an MLP Regressor seems promising:

Train MAE: 301
Train R^2: 0.582
Test MAE: 338 (using cross-validation)
Test R^2: 0.522

My question is how can I improve my results for the prediction (using Python) other than using GridSearch to play around with the MLPRegression function in sklearn? Are there any other models that could be useful in this situation? (I have tried also decision trees, gradient boosting)

In case it is relevant, my dataset contains ~5000 entries since 2008 onwards of individual tenancies, containing: tenancy dates, rent, repair costs, property information and replacements, client information, etc., currently at 41 variables.


texts:
I am building a text-generation model. In the first layer, I am using Word2Vec embeddings. Now since the input is sentences they are variable length and I am padding them with zero. The input is basically zero padded arrays of indexes of the words which appeared. My problem is that 0 is an index of some word, so isn't padding the sequences with 0 is feeding wrong information to the model.
PFB my code. train_x is the training input, train_y is the final word in my sentence, corpus is the entire corpus of sentences. word2idx function fetches index for the word it is passed, max_sentence_len is the max sentence length in the corpus and is used for zero-padding.

train_x = np.zeros([len(corpus), max_sentence_len], dtype=np.int32)
train_y = np.zeros([len(corpus)], dtype=np.int32)
for i, sentence in enumerate(corpus):
    if len(nltk.word_tokenize(sentence)) > 2:
        for t, word in enumerate(nltk.word_tokenize(sentence)[:-1]):
            if word in word_model.wv.vocab:
                train_x[i, t] = word2idx(word)
        train_y[i] = word2idx(nltk.word_tokenize(sentence)[-1])

To elaborate further say the index of hi is 0 and there is 1 and my max_sentence_len is 5 so for an input hi there, my input array looks like [0,1,0,0,0] which translates to [hi, there, hi, hi, hi]. This sounds incorrect to me. Is this how it should be or is there a way to go about it.


texts:
I'm currently working in text mining ptoject I'd like to know once I'm on vectorisation. With method is better.

Is it Word2Vec or Tf-Idf ?
Here I see we can combine them why that? Does it improve quality of data?
What about GloVe?

Thanks


texts: Is K-NN applicable for binary variables?

texts:
I need help because I'm just new to machine learning and I do not know if k-nearest neighbors algorithm can be used to identify the appropriate program(s) for Student 11 in the table below.

The school subjects (Math, English, etc.) are the features, while the 'Program' column has the class labels A, B, and C.
The binary values represent the interest of a student in the subjects.
Can K-NN algorithm find the similar students based on the binary variables in the table?


texts:
What's immediate superset?

Why is it called immediate?

They appear in the context of Apriori algorithm. Particularly the one enhanced with hashing.


texts: A/B testing: How to calculate p-value on post test segments?

texts:
My question on A/B testing is about doing post test segmentation analysis.

For example:

  I run an A/B test on my website to track bounce rate.  On the
  treatment group, i put a video to explain my company. On the control
  group i put just plain text. I pick a segment of users who are first
  time users from USA to be split 50/50 into the 2 groups.

Metric that i am tracking is average bounce rate (assume 20%).
Power effect (0.8)
effect size i expect to see(10% so bounce rate should fall to (20% - 0.10 * 20% = 18%))
Calculated sample size required is say 1000 for each group.

Say i run the test for the correct amount of time. At the end of the test, i get a p-value of 0.06. i do not reject the null hypothesis.

  However, when i do post test segmentation analysis, for example, i saw
  that users who signed up for a free trial, 44% of them played the
  video.

In this case, how do i calculate if the 44% was significant? (while taking into account the multiple comparison problem?)
Like in the Airbnb experiment, they did post segmentation analysis on the browser type and was able to calculate the p-value.

My approach

Does this mean that for every segment that i want to analyze, i need to have at least 1000 samples? Also how would i recalculate the p-value given that the p-value of this A/B test was already generated above as 0.06?


texts: Visualizing N-way frequency table as a Decision Tree in R

texts:
I have a N-way frequency table generated from a regression model fit. This is a reproducible example of such table:

data("CO2")
lm.fit = lm(uptake ~  Type + Treatment, data = CO2)
lm.fit$coefficients

test = count(CO2, c('Type','Treatment'))
test$res = predict(lm.fit, newdata = test)
test$freq = NULL

I am trying to visualize test as a decision tree with nodes as Type and Treatment and res as leaves. I would interpret it as the path the regression model takes, leading to the final value for a particular segment.

I am not able to generate a tree with test. I am also open to other novel ways of visualizing these results. My original problem has many categorical variables, so I am looking for a customizable visualization, something from party::ctree or rattle::fancyRpartPlot.


texts: Theoretical basis for neural network "effort"

texts:
I might be in danger of having my question closed as "not clear what I'm asking for," but here goes.
Suppose we have a simple feedforward network. It has a few layers, each layer has a "reasonable" number of neurons, nothing complicated. Let's say the output has size $n$, and there is no final activation function on the output.
The network will have an "easier" time training to produce some outputs relative to others. In particular, outputs close to 0, that is, closer to the origin in the $\mathbb{R}^n$ output space, will be easier. But this is an intuition I have; I'm not sure if this is actually true. (By "easier" I think what I really mean is in fewer iterations.)
I haven't found a source for this, but there is a lot of advice on the internet to normalize one's data which seems to have a similar motivation.
Is there any theoretical basis for this notion of the effort a network needs to produce an output? Is it meaningful at all to talk about without making assumptions about the function we're trying to learn?
And if this idea is accurate in some way, is the "effort" independent of direction? Is it like an $n$-variate Gaussian distribution where the density of that point corresponds to the effort the network has to produce that point? Or is the "distribution" spikier, making it easier to output e.g. $[0,0,0,1]$ relative to $[\frac{1}{2},\frac{1}{2},\frac{1}{2},\frac{1}{2}]$? (Or vice versa?)


texts: How to pass noisy images as input and original images as labels in Keras - Autoencoders

texts:
I want to make denoising autoencoder, I've added some noise to images, and i want to use them for training, original images will be used as label.
Reading the keras documentation I've found that I need to make generator, since my dataset is large. The function flow_from_directory method from keras documentation is good fit. But I have found one problem.
class_mode: 'input' option from flow_from_directory(directory): method is saying:  images identical to input images (mainly used to work with autoencoders)
My question is: 'How can I pass as input noisy images and then pass as labels original clean images'?


texts: Correcting high AR(1) coefficients in dynamic Gordon model

texts:
I have just finished my thesis on a heterogeneous dividend expectations model applied to the COVID-19 crisis! However after receiving some feedback there is one last issue I want to resolve. I'm using a dynamic Gordon model where estimated AR(1) coefficients (rho and tau) determine variable price-dividend ratios instead of one static value. See the references below for more information on this model.
The problem is that some of my estimated AR(1) coefficients are very high (close to 1) which results in very volatile and unrealistic price-dividend ratios. To pragmatically solve this issue I have corrected these high coefficients by subtracting 1/100 of themselves like this:
$\rho_{new} = \rho_{old} - \frac{1}{100}\rho_{old}$.
This works very well (price-dividend ratios take on realistic values with only a small correction) but it is not very elegant and scientific. Therefore my supervisor asked me to look into this a bit more but I haven't been able to find any related literature on the problem. Does anyone know how to solve this issue in a more elegant way? Any tips and tricks are very welcome!
References:
Hommes, C., & Veld (2017). Booms, busts and behavioural heterogeneity in stock prices. Journal of Economic Dynamics and Control, 80, 101–124
Boswijk, H. P., Hommes, C. H., & Manzan, S. (2007). Behavioral heterogeneity in stock prices. Journal of Economic dynamics and control,31(6), 1938–1970, Appendix B1


texts: Dataset containing spatial and temporal features (built on a CNN model)

texts:
A dataset contains spatial and temporal features.

It contains the time series data (2 min intervals) of the sections of a map.

It is 320*480 (320 map sections and 480-time intervals). Each row belongs to a section and each column is a time interval. So each cell in the dataset is a value of data in a map section in a time interval.

I would like to make a model to import all map sections in 40 min and predict the following 10 next min of all map sections.

So the input shape equals 320*20 and the output shape will be 320*5.

I split data to 320*20 as X and 320*5 as Y samples.

The input and the output of the model are 2-dimensional matrixes.

I can consider the input as a one channel image (320*20*1) and it is possible to use a CNN model.

The codes of the final architecture of my model:

model = models.Sequential()
model.add(Conv2D(256,(3, 3),
                 activation='relu',
                 input_shape=(320,20,1), padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2)))

model.add(Flatten())
model.add(Dense(1600))

model.summary()

The Loss value(MSE) is about 70.
Is it lower than the random prediction?

I reshaped the output to 1600 (320*5) which the final Dense layer expects , while the output is 2d. Does it make the model more prone to error ?


texts: Why do we scale down images before feeding them to the network?

texts:
I have been seeing a lot where images are generally scaled down to either $64\times64$, $32\times32$ or other lower resolutions. Can someone please help me with this and answer a few questions:

Don't we lose image details by doing so?
What would be the consequences if we scale down the image to some other higher resolution such as $512\times512$ or $1024\times1024$ or something?
Can we feed the network without $1:1$ square images?