texts:
I studied the perceptron algorithm and I'm trying to prove the convergence by myself. However, I'm wrong somewhere and I am not able to find the error.

Assumption:

We assume that there is some $\gamma > 0$ such
that $$y_{t}(\theta ^{*})^{T}x_{t} \geq \gamma $$  for all $t = 1, \ldots , n$. The additional number $\gamma > 0$ is used to ensure that each example is classified correctly with a finite margin.
We will assume that all the (training) images have bounded
Euclidean norms, i.e., $$\left \| \bar{x_{t}} \right \|\leq R$$ for all $t$ and some finite $R$

By hypothesis the learning rule is:

$$\theta ^{(k)}= \theta ^{(k-1)} + \mu y_{t}\bar{x_{t}}$$

Now, $$(\theta ^{*})^{T}\theta ^{(k)}=(\theta ^{*})^{T}\theta ^{(k-1)} + \mu y_{t}\bar{x_{t}} \geq  (\theta ^{*})^{T}\theta ^{(k-1)} + \mu \gamma $$
so , by induction
$$(\theta ^{*})^{T}\theta ^{(k)}\geq k\mu \gamma $$

At the same time,
$$\left \| \theta ^{(k)} \right \|^{2} = \left \| \theta ^{(k-1)}+\mu y_{t}\bar{x_{t}}  \right \|^{2} = \left \| \theta ^{(k-1)} \right \|^{2}+2\mu y_{t}(\theta ^{(k-1)^{^{T}}})\bar{x_{t}}+\left \| \mu \bar{x_{t}} \right \|^{2} \leq \left \| \theta ^{(k-1)} \right \|^{2}+\left \| \mu\bar{x_{t}} \right \|^{2}\leq \left \| \theta ^{(k-1)} \right \|^{2}+\mu ^{2}R^{2}$$

So, by induction

$$\left \| \theta ^{(k)} \right \|^{2} \leq k\mu ^{2}R^{2}$$

We can now combine parts 1) and 2) to bound the cosine of the angle between $\theta^∗$ and $\theta(k)$:

$$\cos(\theta ^{*},\theta ^{(k)}) =\frac{\theta ^{*}\theta ^{(k)}}{\left \| \theta ^{*} \right \|\left \|\theta ^{(k)}  \right \|} \geq \frac{k\mu \gamma }{\sqrt{k\mu ^{2}R^{2}}\left \|\theta  ^{2} \right \|}$$

Since cosine is bounded by one, we get:

$$k \leq \frac{R^{2}\left \|\theta ^{*}  \right \|^{2}}{\gamma ^{2}}$$

The problem is that the correct result should be:

$$k \leq \frac{\mu ^{2}R^{2}\left \|\theta ^{*}  \right \|^{2}}{\gamma ^{2}}$$


texts: Generalised data science approach to astronomical catalogue cross-match

texts:
I'm interested in performing a cross-match between two source catalogues in an astronomical context. I'll try and explain the context simply.

A truth catalogue of sources (each with some position, shape and flux properties) is used to create a synthetic astronomical image, by convolving with noise and instrument artefacts.

Then 'some process' is developed (by whatever means - it could be a deep learning algorithm, or it could be a person picking sources manually) which identifies sources in the synthetic image, obviously with no knowledge of the truth catalogue.

The challenge is to map the sources in the 'estimated' catalogue with those in the truth catalogue, after which it is possible to devise a measure of the efficacy of the source recovery approach. Bear in mind that estimated positions/fluxes will differ from the truth due to noise, and not all sources may be detected.

One possible way to do this is to use a positional cross-match algorithm like this: https://arxiv.org/abs/1611.04431, and then refine the possible match candidates down by taking the 'closest' in some multi-dimensional parameter space.

However, I thought there could very well be a more generalised maximum-likelihood estimation algorithm suited to this kind of matching task. I think a nearest-neighbours type routine sounds promising, though I'm unsure how this would scale with hundreds of thousands or millions of sources. If anyone has any suggestions for a starting point, I'd be very grateful.


texts: value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame

texts:
How Can I save a DataFrame as a CSV in HDFS?

I'm trying with this:

df.groupBy("Product_ID").agg(collect_list($"Stock")).saveAsTextFile("PATH/results.csv")


texts: Grouping the Input Features for LSTM (keras)

texts:
When I have a input feature of 2-dimension (variable*feature), is it still good to flatten them into 1-dimension input ({variable*feature}) in order to make a 3-dimensional input (sample,timestep,feature) for LSTM in keras?

Especially, wouldn't it cause a problem if the variables are considered as certain groups?

Assume I have 100 cities and 10 features (population, GDP, employment, living cost, ...) in each city, and then want to try predicting the population of one city. If I flatten the input feature, it would look like:

Time   POP1   GDP1   EMP1   LVC1  ...  POP2   GDP2   EMP2   LVC2  ...
   1  10000   1000   2000   1500      15000   2000   3500   2000  ...
   2  12000   1200   1800   1600      16000   2100   3600   2100  ...
   3  13000   1300   1900   1700      18000   2200   3700   2250  ...
   ...

However, intrinsically, the features in the same category (POP1, POP2, ...) and in the same city (POP1, GDP1, EMP1, ...) will have a strong relationship than each others. Given this, it seems for me that flattering the input feature will lead to omit this implication from the model.

Then,

Is it totally fine to flatter the input feature in this kind of prediction, where there is a group of features like a node in one graph network?
If it is fine, why?
If it is not, what would be a better way to represent this relationship between the cities? (I know Convolutional LSTM would be one solution, but it seems mainly for a larger 2-dimensional input such as images.)


texts: XGBoost: is increasing gamma same as feature selection by average gain?

texts:
Since gamma limits splits unless they meet a minimum gain threshold, isn't that the same thing as removing features that have low average gain? Both will results in splits with higher average gains.
I currently have a dataset with 22m samples, and 200 features. My problem is overfitting, where the model performs well in-sample but poorly out-of-sample. I have improved the model (ie. lowered the overfitting, but not completely remove it) by setting gamma = 1. I am wondering if additional out of sample performance improvements would be possible by lowering the number of features?
My thinking is that removing highly correlated features (R >0.9) will help make the importance of each feature more correct by avoiding dilution of correlated features... but not directly help improve the performance of the model. However, maybe removing low gain features somehow directly improves the model since the algorithm could "use" correlated features to overfit?


texts: Applying the same changes to the test set

texts:
I'm busy working through Aurélien Géron's book. (Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow)
The idea is to split the data into train and test set as early as possible in order to avoid data snooping bias. Afterwards changes are made to the data.
My question is that since changes were made to the training set, I assume the same changes(dropping columns, filling NA rows, converting categorical to numerical, etc) should be made to the test set before training and evaluating? If that is the case, what is the correct way to perform this? Write everything as a function and run it on both, which seems a bit counter intuitive to working with notebooks? Is there a built-in function that I'm not aware of?


texts:
I've trained a neural network network that given a minmax normalized input, provides a minmax normalized output.

this might be late, but is it possible from a minmax normalized output to create the actual output, given you know the actual min and max value?

so an unormalized output?


texts: I am working on a project for prediction, using Fuzzy Cognitive Maps. I am new to data mining and want to learn about Fuzzy Cognitive Maps and its implementation. I also want to know what tools are required for this. I am beginner in this domain, so I need a little help to get started. I would be very grateful for help with some good resources and advice.


texts: Get specific object dimensional position from image

texts: I need to get specific object like logo dimensional position from an image. I checked open-cv, pillow package etc but it does not help to me. Can anyone advice which is the best process to do this? Do we need deep learning for it or we can do it any other way.


texts: Using TF-IDF for feature extraction in Sentiment Analysis

texts:
I am working on sentiment analysis for twitter data, for which I have used Vader to get an approximation of sentiment for a tweet. Along with, I have used TF-IDF for feature extraction. These feature words I am using to train and test a Random Forest model. In my dataset, there are around 3K plus tweets from which I extracted around 570 unique feature words using TF-IDF. And all these features I have used for training the Random Forest Model.

My query is in regards to the usage of this trained model in the real world. What if the new tweets which this model has never seen do not have feature words that I have used for training, will the model fail to make correct predictions for them (in my case there are only 3 possible predictions viz. positive, negative and neutral) correctly for them? If yes, then how should I handle this scenario?

Please let me know in case I am missing something or doing something wrong here.


texts: preparing time series data for building a rnn

texts: I am preparing time series data for to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a compressor along with the time stamps. Like this there is data for around 20 parameters recorded along with their time stamps. Problem is there is differnece in the time stamps at which data is collected. So how do I ideally match the time stamps to create a single dataframe with all the parameters and a single time stamp


texts: How to convert longitude and latitude in time series data from daily to weekly?

texts:
I have time series data like this:

date
longitude
latitude

01/01/2010
-5.42766
107.5784

02/01/2010
-6.42728
104.5245

07/01/2010
-7.42702
105.5816

14/01/2010
-4.42728
99.57834

17/01/2010
-6.41523
103.5562

...
...
...

31/12/2013
-4.42728
99.57834

This is location data (longitude and latitude).
I want to change the data from daily to weekly, something like this:

week
longitude
latitude

week 1
...
...

week 2
...
...

week 3
...
...

week 4
...
...

how to transform the data? since calculating the mean of the data doesn't make sense because it is location data.


texts: Are view() in Pytorch and reshape() in Numpy similar?

texts:
Are view() in torch and reshape() in Numpy similar?

view() is applied on torch tensors to change their shape and reshape() is a numpy function to change shape of ndarrays.


texts: Any guidance for new beginners interested in data science

texts:
I am a student with a master degree in biostatistics. I am interested in data science. I know SAS and R. No experience with python. May I ask for your experts' advice on how to teach myself data science from zero, please?

Any advice is much appreciated.


texts: Ridge regression model creation using grid-search and cross validation

texts:
I created python code for ridge regression.For that I used cross validation and grid-search technique in together. i got output result. I want check whether my regression model building steps correct or not? can some one explain it?

from sklearn.linear_model import Ridge
ridge_reg = Ridge()
from sklearn.model_selection import GridSearchCV
params_Ridge = {'alpha': [1,0.1,0.01,0.001,0.0001,0] , "fit_intercept": [True, False], "solver": ['svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']}
Ridge_GS = GridSearchCV(ridge_reg, param_grid=params_Ridge, n_jobs=-1)
Ridge_GS.fit(x_train,y_train)
Ridge_GS.best_params_

output - {'alpha': 1, 'fit_intercept': True, 'solver': 'cholesky'}

Ridgeregression = Ridge(random_state=3, **Ridge_GS.best_params_)
from sklearn.model_selection import cross_val_score
all_accuracies = cross_val_score(estimator=Ridgeregression, X=x_train, y=y_train, cv=5)
all_accuracies

output - array([0.93335508, 0.8984485 , 0.91529146, 0.89309012, 0.90829416])

print(all_accuracies.mean())

output - 0.909695864130532

Ridgeregression.fit(x_train,y_train)
Ridgeregression.score(x_test,y_test)

output - 0.9113458623386644

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.


texts:
I've looked and surprisingly have not found too much discussion on the relative strengths of the caret and e1071 package.  From my understanding, these packages perform many of the same ML algorithms.  With that in mind, I'm interested in what those practitioners who have experience with both think these packages relative strengths and weaknesses are.  What do you think about readability of code, ease of coding, methods, speed, etc...

In particular I'm interested in multi-class SVM


texts: Customer Segmentation and Category association

texts:
I have to solve two questions on the following dataset:
1. arrange customers into mutually exclusive groups.explain the clusters.
2.identify 1-1 product category association rules for each cluster, i.e. if a customer bought from this category, they are likely to buy from this category too.


texts: Get latest Item by Date for a Recommender System

texts:
I am building a Recommender System where I am giving the User 3 Recommendations depending upon for the Webpage he is on. Let's say My model gives me 3 Recommendations from 2020, 2019, 2015. I would first recommend the User the 2020 article and so on. Currently what I am doing is just filtering the JSON Recommendations from my Dataframe as the Latest articles. Is there any Scalable and efficient way to do this.
[('https://www.webpage.tv/,
  '2019-06-04T00:00:00'),
 ('https://www.webpage.tv/',
  '2019-08-30T00:00:00'),
 ('https://www.webpage.tv/',
  '2020-03-02T00:00:00')]

output_df['latest'] = output_df['recommendations'].apply(lambda x : sorted([(value,key) for key,value in x.items()], key=lambda x:x[1]))


texts:
I am working on making predictions about traffic trends in a big city. Traffic is very seasonnal, and depends on the time of the day (with peaks at 9am and 6pm), day of the week (Monday to Friday are alike compared to Sat and Sun), holidays (Xmas, school holidays etc). And I have 2 years (at hourly rate) of data.

I am using a RandomForestRegressor to make a prediction on the year to come. Is that a correct algorithm for this class of problem?


texts:
I've been looking at autoencoders for disparate uses such as dimension reduction, blurring or sharpening images and data denoising.

What methods are used to determine acceptable loss levels for autoencoders?


texts: How can I tune my neural network to solve a group of equations?

texts:
I am trying to use neural networks to solve the following group of equations:

$\sqrt{(x_1 - x)^2 + (y_1 - y)^2}$ - $\sqrt{(x_2 - x)^2 + (y_2 - y)^2}$ = $d_{12}$

$\sqrt{(x_1 - x)^2 + (y_1 - y)^2}$ - $\sqrt{(x_3 - x)^2 + (y_3 - y)^2}$ = $d_{13}$

Where x and y are unknown variables and x1, y1, x2, y2, x3, y3, d{12} and d{13} are all known.

As I am only interested in a certain range of all the x stuff and y stuff and the x and y do not need to be exactly accurate, I think I can solve the equations by a neural network which is believed to be a function approximator. My idea is randomly generating 50000 data points. Each data point includes whole randomly generated x1, y1, x2, y2, x3, y3, x and y, then I can use the equations above to have d{12} and d{13}. I feed x1, y1, x2, y2, x3, y3, d{12} and d_{13} into a fully connected neural network as input, and train the nets to output x and y.

The architecture of my current networks:

I have 4 hidden layers
The numbers of neurons in each layer are: 128, 256, 512, 1024, 2048, 4096, 4096, 2048, 1024, 512, 256, 128.
The activation functions are all Relu, the output layer doesn't have activation function since it is a regression problem.
In each layer, I did batch normalization.
The batch size is 256, and I train the network for 200 epochs.

Unfortunately, this network doesn't perform well, the error for the output is huge. I suspect maybe the network stucks in some local minima or the function that the neural network wants to mimic is far more complicated. I am wondering if anybody has ideas about why it is that and how to tune it.


texts: I have read quite a lot about discretization techinques, measuring WoE and IV etc., but the basic question - when is it worth to use binning and when isn't - is not entirely clear. On the one hand, the result of discretization seems quite "pretty" and more easy to handle for me, but on the other hand it is always emphasized that binning always causes information loss. Could you provide some clues when should one use this technique?


texts: Python Ludwig: interpretation of final coding metric

texts:
I am learning über ludwig in Python. I ran the yaml definition file. I sat the epochs to 10. I have attached the screen shots of last 4 epochs.
Screen shot 1

Screen shot 2:

After running the program, it displays the best epochs

I am not sure how did it say epoch 7 hastiest validation model. If you look at other epoch, they have better training, vali, and test accuracy then how come it selected epoch 7. Could anyone shed some light on it?


texts: K-modes clustering: Estimating which features were most impactful on clustering?

texts:
I have entirely categorical data (survey results from users), so I've used k-modes clustering to better understand my users.

I'm not an expert at clustering methods at all. Is there a way to known way of estimating the importance of a feature (or combination of features) in deciding which cluster a user falls into?


texts: ImageNet vs CIFAR pre-trained models for biomedical image classification

texts:
I'm doing research on transfer learning for biomedical image classification, mainly skin lesion classification. From what I know, both CIFAR and ImageNet are pre-trained on natural images and not biomedical images. However, which approach would be more ideal for analyzing dermoscopic images?
I apologize if its a naive question


texts: Neural Network regression negative performance

texts:
I have a problem with the performance of a multi layer perceptron regressor (neural network) and I cannot figure out why.
Task: I am trying to improve a time series prediction. I have predictions of a physical parameter of the last 4 years along with the quasi true values. I train the NN with the predictions for -7 days until +1 days around the day I am interested in as features, in order to obtain a better prediction for that day.
Problem: The output of the NN is worse than the feature for the day I am interested in, both for the training and the testing data. Both in terms of RMSE and MAE. I expected the output to be at least on the same level as the feature I input to the NN.

Method: Python with sklearn. I use a grid search with cross validation to get good hyper parameter. I test for different hidden layer configurations, activation functions, learning rate and regularization penalty strengths. I split the data into 66% for training and the remaining data for testing.
I am really grateful for tips how to figure out what my problem here is.
EDIT:
I am using sklearn.neural_network.MLPRegressor which provides ‘identity’, ‘logistic’, ‘tanh’ and ‘relu’ as activation functions and I have teseted all of them in the grid search.
I did NOT scale the feature matrix because all features are in the same unit as the desired output and range from about -1 to +1.
EDIT2:
tuned_parameters = [{'hidden_layer_sizes': [int(2/3*number_features),
                                        (int(2/3*number_features), int(4/9*number_features)),
                                        (int(2/3*number_features), int(4/9*number_features), int(8/27*number_features))],
                 'alpha': 10.0 ** -np.arange(1, 4),
                 'activation': ["identity", "relu", "logistic", "tanh"],
                 'learning_rate': ['adaptive', "invscaling"],
                 'solver': ['lbfgs'],
                 'early_stopping': [True],
                 'max_iter': [600]}]

regr = GridSearchCV(MLPRegressor(), tuned_parameters, n_jobs=3, verbose=2)
regr.fit(feature_training_matrix, combined_training_target_vector)

Data:
The prediction data I use has the following structure: for every day of the last ~4 years there were predictions made for the next 90 days. I have a text file with -90d to +90d data for every day. I try to train the NN to estimate a better prediction for the next 10 days. For this I take -7 up to +1 days around the current prediction day (1-10 days after the currently used starts to predict) as features. This means that the predcition of the day I am interested in is included as a feature.
feature example: [0.16272058, 0.13296574, 0.14213905, 0.25064893, 0.23302285,
   0.21019931, 0.20733988, 0.1466959 , 0.17029025, 0.15876942]

corresponding target: 0.174652


texts: Replacing words by numbers in multiple columns of a data frame in R

texts:
I want to replace the values in a data set (sample in the picture) using numbers instead of words, e.g., 1 instead of D, -1 instead of R, 0 for all other values. How can I do it with a loop? I know it can be done doing this instead:

(suppose d is index name)

d[d$Response == "R",]$Response = -1

d[d$Response == "D",]$Response = 1

... (other values code it and assign value of) = 0


texts:
I need to perform classfication of hundreds of classes. New classes arrive regularly. I also have some large training set (thousands of samples).

OneVsRestClassifier from sklearn trains all its underlying classifiers with the same hyperparameters. My idea is that I can train one classifier for each class separately, when class arrives. This should give the following benefits:

No need to refit all previous classifiers. Given training data is large enough, one additional class doesn't make big difference for what's already trained. Training time doesn't grow linearly on classes.
Hyperparameters are chosen separately for each classifier which should improve accuracy.


texts: Does L1 regularization always generate a sparse solution?

texts: Following is a plot in Bishop's Pattern Recognition and Machine Learning explaining why $L_1$ regularization would lead to sparse solution, and I understand that in this case $w_1$ would be automatically constrained to be 0.  However, it seems to me that it is not always true that the corner point is closest to the optimal unconstrained solution.  For instance, if the optimal unconstrained solution is at the red cross, then there exists some point on the side that is closer to the red cross than corner points.  In this case, do we still have sparse solutions?


texts: Using previous hour's value in time series data for inclusion in random forest

texts:
I have a training data set, which is something like the following:

hour   windspeed   pollution
0              5           8
1              6           9
2              3           1

I am using scipy's RandomForestRegressor to estimate pollution given windspeed, hour. It is doing "ok", with $r^2 = 0.7$ and $rmse = 35$. However, I would like to include the previous hours pollution values as additional predictors, as in:

hour   windspeed   pollution   pollution@t-1  polluion@t-2 ... pollution@t-n
0              5           8              NA           NA
1              6           9               8           NA
2              3           1               9           8

Eventually, I will be applying this to future weather conditions, estimating pollution, where pollution can only be known at time zero:

hour   windspeed  pollution  p@t-1  p@t-2  p@t-3  p@t-4    ...
1              5          ?     34     28     25     ...   ...
2              6          ?     NA     34     28      25   ...
3              3          ?     NA     NA     34      28   ...

How should I best go about including this in a Random Forest? Since RFs don't tolerate null values, how best to impute them? Since '0' is not appropriate (given that 0 is likely very far from the current value), I was thinking that perhaps NaNs could be filled with the nearest known value, as in:

hour   windspeed  pollution  p@t-1  p@t-2  p@t-3  p@t-4    ...
1              5          ?     34     28     25     ...   ...
2              6          ?     34     34     28      25   ...
3              3          ?     34     34     34      28   ...


texts:
What do we refer to as "dense predictions" for time series ?

I am reading through this

  Or to make dense predictions one step into the future, you might shift the features and labels by one step relative to each other:

def dense_1_step(batch):
  # Shift features and labels one step relative to each other.
  return batch[:-1], batch[1:]

predict_dense_1_step = batches.map(dense_1_step)

for features, label in predict_dense_1_step.take(3):
  print(features.numpy(), " => ", label.numpy())

What does this shift between features and labels helps towards dense predictions ?


texts: What is the minimum number of data to create a function?

texts:
I gonna introduce the problematic like this:

Let's say there are individuals with different capacities/skills. These capacities/skills are depending of the environment: nature of the floor, weather temperature, wind speed. The capacity/skill I want to study is the speed of the individuals considering the environment.
As you know, speed is depending of the distance, not only because of the formula distance/time unit, but because even Usain Bolt cannot reproduce the same speed for the 100m and 200m discipline, actually his mean speed decrease with the distance, with environment's variables as constants. This is why I cannot just look at the mean speed itself to conclude if it is more adapted or not
So, because factors are variable/do change, I want to isolate with marginal effect, and conclude what nature of the floor is better for this individual to perform, what distance is better to perform too (Usain Bolt is not done for endurance...). To do it, I want to use a linear regression, this is more simple to derivate.

The main issue is my data has not always a lot of historic. This is often that there are no more than 5-6 previous experiences.

So, when I was younger in high school, there were a rule of thumb telling we need a least 5 points to make a function (with OLS). Know that I am older and have more experience, I doubt of this small threshold.
But I have not enough experience to know what really could be this threshold. Do you have any idea ?


texts: What does it mean the term variation for an image dataset?

texts:
I am working with convolutional neural networks, and I have seen that often we need to pre process the images before feeding them to the network. In particular, I have seen that often we have to do image augmentation using an image generator. Now, when looking for a clarification on why we need to do this, I came across an article which says:

  Image Augmentations techniques are methods of artificially increasing the variations of images in our data-set by using horizontal/vertical flips, rotations, variations in brightness of images, horizontal/vertical shifts etc.

What I don't understand is what is the variation of a dataset.

Can somebody help me? Thanks in advance.


texts: What is a good explanation of Non Negative Matrix Factorization?

texts: I am trying to find a resource to understand non-negative matrix factorization. Apart from Wikipedia, I couldn't find anything useful.


texts:
I know that BPTT is the method to apply Back Propagation on RNN.

Which is works fine with RNN as it stops at certain point as changes approach to zero

but isn't it the exact Vanishing Gradient Problem?

if it is the same then Why does it have 2 names, one is a problem and one is a method.

if not then what am I missing here, what is the difference between them?


texts: How to plot a heatmap-like plot for categorical features?

texts:
I would greatly appreciate let me know how to plot a heatmap-like plot for categorical features?

In fact, based on this post, the association between categorical variables should be computed using Crammer's V. Therefore, I found the following code to plot it, but I don't know why he plotted it for "contribution", which is a numeric variable?

def cramers_corrected_stat(confusion_matrix):
    """ calculate Cramers V statistic for categorical-categorical association.
        uses correction from Bergsma and Wicher,
        Journal of the Korean Statistical Society 42 (2013): 323-328
    """
    chi2 = ss.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum().sum()
    phi2 = chi2/n
    r,k = confusion_matrix.shape
    phi2corr = max(0, phi2 - ((k-1)*(r-1))/(n-1))
    rcorr = r - ((r-1)**2)/(n-1)
    kcorr = k - ((k-1)**2)/(n-1)
    return np.sqrt(phi2corr / min( (kcorr-1), (rcorr-1)))

cols = ["Party", "Vote", "contrib"]
corrM = np.zeros((len(cols),len(cols)))
# there's probably a nice pandas way to do this
for col1, col2 in itertools.combinations(cols, 2):
    idx1, idx2 = cols.index(col1), cols.index(col2)
    corrM[idx1, idx2] = cramers_corrected_stat(pd.crosstab(df[col1], df[col2]))
    corrM[idx2, idx1] = corrM[idx1, idx2]

corr = pd.DataFrame(corrM, index=cols, columns=cols)
fig, ax = plt.subplots(figsize=(7, 6))
ax = sns.heatmap(corr, annot=True, ax=ax); ax.set_title("Cramer V Correlation between Variables");

I also found Bokeh. However, I am not sure if it uses Crammer's V to plot the heatmap or not?

Really, I have two categorical features: the first one has 2 categories and the second one has 37 categories.

I need the plot will be like the two last plots presented here, but also display the association values on it too.
Thanks in advance.


texts:
Im new to ML. I'm trying to predict if a new Music Album will exceed X amount of dollars in Sales.
I'm looking to build a model to go only after potential best sellers. I do have historic data for Music Sales from 2010 till 2016. I have many signals:

Music Genre
Music Band/Artist name
Label
Year released
Country of origin
Part of a Series/Volume... etc.
Sales per month

What type of ML problem is this one?


texts: How to properly store a list of pandas dataframes, so that I can import them in a new python script?

texts:
I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index).

In this GUI, I want to have Radiobuttons to tell the program which list to use, and an Input to tell the program which one of the dataframes in the corresponding list to show. Now, I have made a crude algorithm (without a GUI) to do this, and it works, however the lists and, thus, the dataframes, are stored as variables in the original Jupyter notebook, which is full of the convoluted and inelegant code I used to process and store these results.
I managed to save them using %store, and was able to use them in a new Jupyter notebook, but for the GUI I will need to use a separate Python script file. My problem is that this command doesn't work when I try to use it in the script file, to restore and use the variables there.

tldr: Is there a proper way to store lists of dataframes, exactly as they are (I don't want the indexing to be messed up), so that I can call them/import them in a separate Python script, and use them in the GUI?

P.S.: As you can understand, I'm new to Python, so I'm sorry if my programming methods seem inelegant and my question trivial.


texts: Explanation for Why Logistic Regression can be so Accurate in Sentiment Classification?

texts: My question is about how a logistic regression model performs so accurately. In some exploratory experimentation, I compared a logistic regression model against a long short term memory recurrent neural network for binary classification of consumer reviews (Positive or Negative)(25,000 reviews). The Logistic Regression model performed better with an accuracy of 88.0% and a recall, precision, and f-score of 0.88. The Recurrent Neural Net performed with an accuracy of 81.1%, a recall of 0.83, a precision of 0.80, and an f-score of 0.81. Perhaps my building of the network was flawed but I was still surprised to see that the Logistic Regression had such a high accuracy.


texts: how to use standardization / standardscaler() for train and test?

texts:
At the moment I perform the following:

estimators = []
estimators.append(('standardize', StandardScaler()))
prepare_data = Pipeline(estimators)

n_splits = 5
tscv = TimeSeriesSplit(n_splits = n_splits)

for train_index, val_index in tscv.split(df_train):
    X_train, X_val = prepare_data.fit_transform(df_train[train_index]), prepare_data.fit_transform(df_train[val_index])

X_test = prepare_data.fit_transform(df_test)

Now I would like to know if this is correct. My concern is that X_train and X_test are transformed separately. While in the first instance I thought this is how it should be I'm about to change my mind as I think I have to use the mean and std of the train set to use within the test set?


texts: xgboost Predictions from R and Python don't match

texts:
Thought maybe somebody here could help us solve a mystery (https://github.com/dmlc/xgboost/issues/1623):

We are trying to build a xgboost prediction function in R for a model that was trained in Python and the results don't match. See below for an example of how to reproduce.

Fairly new to xgboost, particularly using it across languages, so may be missing something obvious.

Steps to reproduce:

(1) Download this model file: http://ml.stat.purdue.edu/hafen/WTKG.model

(2) Run this script in R:

library(xgboost)
mod <- xgb.load("WTKG.model")
x <- c(91, 9, 9, NA, NA, 273, 20, 170, NA, NA, 14, 14, 0,
  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,
  0.95490834613, 1, 90, 0.95490834613, 1, 90,
  0.93087973569, 357, -266, 0.93087973569, 357, -266,
  0.95490834613, NA, 0.93087973569, NA, NA, NA)
d <- xgb.DMatrix(matrix(x, nrow = 1), missing = NA)
predict(mod, d)
# [1] 0.6483372

(3) Run this script in Python:

import numpy as np
import xgboost as xgb

bst = xgb.Booster({'nthread': 4})
bst.load_model('WTKG.model')
x = [91, 9, 9, np.nan, np.nan, 273, 20, 170, np.nan, np.nan, 14, 14, 0,
  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,
  0.95490834613, 1, 90, 0.95490834613, 1, 90,
  0.93087973569, 357, -266, 0.93087973569, 357, -266,
  0.95490834613, np.nan, 0.93087973569, np.nan, np.nan, np.nan]
d = xgb.DMatrix(data=[x], missing=np.nan)
bst.predict(d)[0]
# 1.3775804


texts: Why use different variations of Softmax in training and validation for neural networks with Pytorch?

texts:
Specifically, I'm working on a modeling project, and I see someone else's code that looks like

def forward(self, x):
    x = self.fc1(x)
    x = self.activation1(x)
    x = self.fc2(x)
    x = self.activation2(x)
    x = self.fc3(x)
    x = self.activation3(x)
    # use log softmax + NLLLoss in training; softmax to make predictions
    if self.training:
        x = self.log_softmax(x)
    else:
        x = self.softmax(x)
    return x

For context, this is using PyTorch, and it is on a classification problem. The criterion is NLLLoss.
What's the rationale behind using log_softmax  for training but using softmax for actual predictions?


texts: For an upcoming project, I'm mining textual posts from an online forum, using Scrapy. What is the best way to store this text data? I'm thinking of simply exporting it into a JSON file, but is there a better format? Or does it not matter?


texts:
I am trying to build a NN which can predict exchange values. I am quite new to R and NN and I don't quite understand how I could improve the performance metrics of the neural network. I have tried adjusting the threshold and learning rate as much as I could before it stopped improving the result. Can I improve the result further and at what point do you stop and accept the result?
I have added a link to the datafile because some code works on the smaller snippets of the dataset but sometimes don't work in the full dataset
Dataset: https://www.dropbox.com/s/17exy1968lsidsc/ExchangeUSDcsv.csv?dl=0
Code:
library(neuralnet)
library(grid)
library(MASS)
library(ggplot2)
library(reshape2)
library(gridExtra)
library(neuralnet)
library(zoo)

normalize <- function(x){
  return ((x - min(x)) / (max(x) - min (x)))
}

exchangeData <- read.csv("ExchangeUSDcsv.csv")
data <- exchangeData$USD.EUR
data <-as.data.frame(data[!is.na(data)])

#delay the data
currencyDelay <- embed(data[[1]], 4)[, 4:1]

#delay and normalize
currencyNorm <- embed(normalize(data[[1]]), 4)[, 4:1]

currencyNorm <- as.data.frame(currencyDelay)

training <- currencyNorm[1:400,]
testing <- currencyNorm[401:497,]

set.seed(1234)

currency_model <- neuralnet(V4 ~ ., hidden = c(5), data = training, act.fct = "logistic")

#Get the model to predict
model_results <- compute(currency_model, testing)
#Store models predictions
predicted_result <- model_results$net.result

#Get original train and test data
currencyDelay_train <- currencyDelay[1:400,]
currencyDelay_test <- currencyDelay[401:497,]

#Find min and from the data
cur_min <- min(currencyDelay_train[,4])
cur_max <- max(currencyDelay_train[,4])

#reversed normalization
unnormalize <- function(x, min, max){
  return ((max-min)*x + min)
}

#Un-normalize the predictions
models_predictions_unnorm <- unnormalize(predicted_result, cur_min, cur_max)

#Stat Indicies
rmse(currencyDelay_train[4], models_predictions_unnorm)
mae(currencyDelay_train[4], models_predictions_unnorm)
mape(currencyDelay_train[4], models_predictions_unnorm)

#Compare Actual results with NN prediction
final_result <- cbind(currencyDelay_train[4], models_predictions_unnorm)
head(final_result)

Current outputs:
RMSE:
0.04519089
MAE:
0.04519089
MAPE:
0.03294277


texts: When using KNN, how do I know which order of the Minkowski distance to use?

texts: I am learning about KNN and ML in general. I know that KNN usually uses second order Minkowski distance (Eucledian Distance), but I assume it cal also use other orders. But what is the benefit to choosing a higher order with respect to the general performance of the model? Is it faster? More accurate? Does the level of noise influence my decision for the order? How do I know that because of this and this condition, I have to use a higher order Minkowski distance?


texts: KL divergence for time-series prediction

texts:
I am training a model to predict time series data. I would like the residuals of the model to be approximately normally distributed, as this is convenient for making probabilistic statements.

The model I'm considering works as follows, with i/o scaled to into the same range:

               Batch of observations ↴
Batch of inputs ⟶ Batch of outputs ⟶ Batch of residuals ⟶ Bins

The model is then trained using KL divergence so that the bins approximate a (discrete) normal distribution on the batch.

What I'm having trouble with is the binning part. I need a layer where the activation of each node is roughly the proportion of samples in the batch that are in each bin. That is to say, if there are 100 samples in the batch and 15 of them result in a residual between -0.1 and 0.1, then the activation of the the "[-0.1,0.1] neuron" should be close to 0.15.


texts: What is the difference between HLC (Histogram of local features) , CSS ( color self-similarity) ans MDST (Max DisSimilarity of Different Templates)

texts:
I'm new to computer vision and have been researching for Master thesis purposes in Detection algorithms and the techniques used in each. As I arrived to the point where alot of papers showed the importance of color in object recognition, i got got bumped with HLC MDST and CSS. So my question is : are they all literlally a way to describe the distribution of the color in an image? If yes I would be glad for a brief explanation ,what does each of them do, and a dedicated source link for that is available would be great.
The three methods are briefly viewed in Qingyuan Wang1's work."Justifying the importance of color cues in object detection:
a case study on pedestrian"
Thanks in advance


texts: Accuracy drops if more layers trainable - weird

texts:
I'm training a 4-class classifier. I load pre-trained weights from vgg16, and only set the last layer to be trainable. This gives me a loss (categorical crossentropy) of 0.9 and a training accuracy of 70%.

However, I want features for my images - the penultimate activations is one choice for getting features for every image. Hence I set the last and the penultimate layers to both be trainable, but then my loss jumps to 9.8 and training accuracy drops to 39%.

How can this happen? Here's the code I'm using:

vgg.model.pop()
for layer in vgg.model.layers:
    layer.trainable = False
vgg.model.add(Dense(trn_batches.nb_class, activation='softmax'))
opt = Adam(lr=0.001)

#only the last layer is trainable
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,
    validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

#make last 2 layers trainable (starting from -3 because there is a dropout layer)
for layer in vgg.model.layers[-3:]:
    layer.trainable = True
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,   validation_data=val_batches, nb_val_samples=val_batches.nb_sample)


texts: Distribution of training dataset in binary classification problem

texts: Is it important to have the same amount of positives and negatives in our training dataset for a binary classification problem? Or for example, it doesn't matter if we have 70% positives and 30% negatives.


texts: How can I compare the contribution of two predictors in two different sorts for machine learning algorithms?

texts:
I'm new to machine learning and try to clarify my problem in research. I just wonder if I can compare the importance of two different variables in two different sorts. For example, A and B are two variables that I want to compare their contribution to ML accuracy. The rest of the variables (like C, D, and E) for each sort are the same.

If I got the results that the rest of the variables (C, D, and E) contributes more (obtain a higher importance score) in the model predicted by A and the rest, can I just say B performs better for ML and its contribution is more significant than the model with A and the rest?


texts: Removing the effect of Time series X on time series Y, when their relation is unknown

texts:
I am working on a dataset of 6 years measurements of a water quality parameter called 'chla' ( parameter 'X') measured by a sensor for each year from May to October. The parameter has its own trend and irregular changes by month, day and even hour, but due to some proven biological theories, it also gets an inverse impact from solar radiation (parameter 'Y'), so that a daily periodicity effect can be observed on the data ( parameter 'X'). As a result, there is a periodical behavior in the parameter 'X' that starts from sunrise, maximizes in the noon and minimizes in sunset again and last to next sunrise. However, this effect is not very regular because some days or times are cloudy (low solar radiation).

My goal is to remove the effect of 'Y' on 'X'. I have the time series of both 'X' and 'Y' parameters sampled every 10 minutes. Is there any solution to remove the Y effect on X without knowing their exact mathematical relationship. the image below shows two images of parameters for a short time period.
cyan = parameter 'X' ; red = parameter 'Y'


texts:
Say I have a data set such as the following:

person, Time, Value, Event
person1, 2010-07-02 00:00:00, 5.4, 0
person2, 2010-07-02 10:00:00, 12.7, 0

We have a current model in place at work that doesn't take into account the temporal aspect of our data. In that implementation, the model was trained with only unique values for 'person', and it throws away the time variable. However, it has come to our attention that we can look at our data as a sequence instead. This starting time is unique for each person, and clearly associated with only that person, so merely pretending each person is independent and just treating each row as an individual data point wouldnt make any sense. The following is what I've restructured the data as:

person, Time, Value, Event
person1, 2010-07-02 00:00:00, 5.4, 0
person1, 2010-07-02 00:00:15, 3.6, 0
person1, 2010-07-02 00:00:30, 2.4, 0
person2, 2010-07-02 10:00:00, 12.7, 0
person2, 2010-07-02 10:01:15, 12.8, 0
person2, 2010-07-02 10:01:30, 13.1, 1

This sequence for each person would continue until and 'event' or 'non-event'. I'm totally unfamiliar with machine learning on time series data. All of the examples I've read with different models treat the data as one big sequence corresponding to one entity, while our data clearly doesn't work like that. Is the way I've structured the data the right way to approach a time series model? And if so, what would be an appropriate model to consider?


texts: Extract relevant features from time series data

texts:
I have a time series data set from a sensor and the task is to predict the time before a failure event is occurred. The data set has one feature and has almost 20 million rows. This is a regression problem.

I tried polynomial features, auto correlation, rolling statistics and expanding statistics. The only one that seemed to improve my model was expanding sum. What are some relevant features to be extracted from this data?

My model is a Linear Regression model, the data set was scaled and currently only two features improved my model. The sensor data and the expanding sum. Any other suggestions to tackle this problem other than using deep learning?

Update: For clarification I added the plots for both the input and output.

Sensor reading plot - Input

Time to failure plot - Output


texts: How i match with SequenceMatcher ratio two dataframe

texts:
I use the SequenceMatcher ratio to match two dataframe with the best ratio.
I want to check first if the score A and AA is good then check if the score between B is BB is good then if the score between C and CC is good, then I add the line
I tried this function, but it doesn't work:
from difflib import SequenceMatcher
def similar(a, b):
    ratio = SequenceMatcher(None, a, b).ratio()
    return ratio
order = []
score = []
for index, row in df1.iterrows():
    maxima = [similar(row['A'], j) for j in df2['AA']]
    best_ratio = max(maxima)
    if best_ratio > 0.9:
        maxima2 = [similar(row['B'], j) for j in df2['BB']]
        best_ratio2 = max(maxima2)
        if best_ratio2 > 0.9:
           maxima3 = [similar(row['C'], j) for j in
                      df2['CC']]
           best_ratio = max(maxima3)
           best_row = np.argmax(maxima3)
            order.append(best_row)
df2 = df2.iloc[order].reset_index()
merge = pd.concat([df1, df2], axis=1)


texts: Correlation among variables (categorical, binary and numerical)

texts:
I have the following data
 Prefix Label   Year    Symbols Tags    %_Men   Prop    Uni_Role    Uni_Role_2  #_Cars  #_App
    0   IN  0.0     1996    []        []    0.08    0         11         22           0     1
    1   IT  0.0     2000    []        []    0.00    0         2          46           1     1
    2   ES  1.0     1999    []        []    0.10    0         10         30           0     1
    3   UK  1.0     1999    [!]      [!]    0.07    1         10         25           0     1
    4   USA 0.0     2007    []        []    0.07    0         11         23           0     1
    6   Republic of San Marino 0.0  1996    []  []  0.07  0   10         20           0     1
    7   USA 0.0     2002    []        []    0.05    0         5          22           0     1
    8   IN  0.0     2008    []        []    0.03    0         4          18           0     1
    9   IN  0.0.    1997    []        []    0.04    0         9          19           1     1
    10  IN  0.0     1997    []        []    0.06    0         8          20           3     1

I would like to study correlation among variables (target variable is Label). Since there is a mix of categorical (including lists, under Symbols and Tags columns), numerical and binary variables. For categorical variables probably I should consider Chi-Square, for numerical Pearson. I do not know how to get information from binary, but I know that it should not be possible to apply Pearson.
The problem is a classification one, where I should predict the status of a citizen.
Have you had similar problems? Any idea on how I could do that?


texts: Dimensionality reduction with PCA limitations

texts: What are the cases when we should not use PCA for dimensionality reduction and what to use in such cases?


texts: propensity to pay in healthcare Datasets

texts:
I am searching for datasets or data sources (free or paid) to do a propensity to pay for healthcare industry.
I found many solutions that offer complete solutions but whats interesting to me that they always include 3rd party data sources. I wanted to know where I can find such data, and where also can I ask.


texts: Are there any algorithms for a classification problem involving unlimited classes, and only a few instances per class

texts:
The Scenario: A group of people must summarize specific parts of speeches they hear. They hear a new speech every day, and it's possible that multiple members of group are listening to the same speech.

The Question: What ml algorithm or approach could you take to link summaries that refer to the exact same part of the exact same speech?

Why this question is interesting: most classification problems deal with a large amount of instances and only a few classes. This problem is about dealing with very few instances (1-5) for a growing number of classes (The specific part of the specific speech).


texts: How to properly train CNN on Full Digital Mammography

texts: I am trying to train my Convolutional Neural Network on full digital mammography images. Here are example dimensions from a random sample: (5832, 4104, 3). Which method for loading images and training the model is recommended in this case? I guess I would need to load only certain amount of images in memory so that it does not overflow. Should I use Keras ImageDataGenerator or TensorFlow's TFRecords?


texts: Tuning parameters in Affinity Propagation

texts:
I am doing Affinity Propagation clustering and trying to do tuning, but it takes time. A lot of time actually. As I am beginner I do not know how to get clusters. I need cluster numbers from 1 to 20 for example. Without any evaluation.

from sklearn.metrics import silhouette_samples, silhouette_score
from sklearn.cluster import AffinityPropagation

final_n_clusters = []
preference = np.arange(-1,1,0.1) # HERE
iter_value = np.arange(1,10,1)   # and HERE
iter_value = np.array(iter_value, dtype=np.int32)

for k in preference:
    n_cluster_list = []
    for j in iter_value:
        af = AffinityPropagation(preference = k, max_iter = j, affinity='precomputed').fit(X)
        labels = af.labels_
        n_clusters = len(np.unique(labels))
        n_cluster_list.append(n_clusters)
    final_n_clusters.append(n_cluster_list)


texts: What's the best classification model for this recommendation engine?

texts:
I'm not a data scientist but I'm trying to implement a recommendation engine on my company. My application runs on PHP but I'll use Python to process this data.

My company is an online school, with 40 online courses as of now. I have a CSV file with around 30k users preferences, and it looks like this:

0 means that user is not subscribed (I consider here that he has no interest), while 1 means subscribed (interested).

My idea is to compare one single user array such as [0,1,0,0,0,1,1...] with all this data and return a grade for each course with the probability of interest for this user.

I was thinking of using a Multinomial Logistic Regression, but as far as I know (and I don't know much) it would return me a binary result, right?

What classification model would you recommend me to use? Ideally, my result should be something like:

[0.95, 0.1, 0.54, 0.3, 0.87...]

Cheers!


texts: Can the Generative Adversarial Network useful for Outlier detection and Outlier explanation in a high dimentional numerical data?

texts:
I have been building a model to find explanation of Outliers in a high dimensional numerical data, generated from many sensors. The data contains more than 350 different fields and each field has numerical values (Float or Integer). It looks like: 350 columns and many rows. I want to find the outliers/ anomalies in the data and also the Explanation why those values are outliers.

I was reading about Generative models and found that "They have the potential to understand and explain the underlying structure of the Input data even when there are no labels." I would like to know if it will be good to use the GANs for outlier detection and explanation on numerical data?


texts: Is it better to have one model with more categories or less with two for multi-label classification?

texts:
For classifying text into three classes question, complain and  complements
where each sample can have multi-labels (question and complains,  question and complements):

is it better to have one model for all three targets?
or two models, the first for (question or not) and the second one for (complains, complements or else)?

which approach is better when the data are labeled, unlabeled and unbalanced?


texts:
In a dataset consisting of 1,000 samples, it has been shown that a 70-30 split
(i.e. 70% of the samples used for training, 30% for validation) will provide a good estimation
of the test accuracy of the trained models. If the dataset size increases to 10,000 samples, what
split would you suggest?


texts: collaborative filtering using graph and machine learning

texts:
What are the advantages and disadvantages of using Collaborative filtering based recommendation using machine learning approach and graph based approach ?

Say I have user purchase data (user_name, user_location, user_company_name, product_name, product_price, product_ingredients) and would like to recommend product for user based on what other user from the same location, company are buying, based on product price, ingredients etc.

How to decide on which of them is suitable for a given use case?
I would like to evaluate Neo4j (Graph database) and Mahout (Machine learning) for Collaborative filtering.


texts: Correct theoretical regularized objective function for XGB/LGBM (regression task)

texts:
I am writing an academic paper on the application of Machine Learning methods to Time Series Forecasting and I am unsure about how to write down the theoretical part about the regularized objective function for XGBoosting. Below you can find the equation given by the developers of the XGB algorithm for the regularized objective function (equation 2). The paper is called "XGBoost: A Scalable Tree Boosting System" by Chen & Guestrin (2016).
In the Python API from the xgb library there is a way to end up with a reg_lambda parameter (L2 regularization parameter; Ridge regression equivalent) and a reg_alpha parameter (L1 regularization parameter; Lasso regression equivalent). And I am a bit confused about the way the authors set up the regularized objective function. According to my understanding the ridge penalty is given by $\lambda \sum_{j=1}^p \beta_j^2$ and the lasso penalty is given by $\lambda \sum_{j=1}^p |\beta_j|$. However, the authors of the paper seem to use a combination of the two things that is not equivalent to either one since they apply the absolute value and square the beta term.
My questions are therefore the following ones:
(a) How does the regularized objective function look like if you want to allow for the inclusion of a l1 and a l2 parameter in the same model?
(b) How does the objective function of the LGBM(light GBM) algorithm differ from XGB or is it identical since the developers of the LGBM algorithm do not provide any theoretical details about the objective function or the iterations of boosting in their paper "LightGBM: A Highly Efficient Gradient Boosting" (Ke et al., 2017). According to my understanding, the only difference is the computation which is faster since the LGBM algorithm computes the information gains on the regression tree splits more efficiently and the features are bundled or am I misinterpreting/overlooking any aspects here?
(c) does it make sense to include both a l1 and a l2 regularization term in the same boosting model from a theoretical and practical perspective?


texts: An CNN seems like capturing specific range of input data. (Image Segmentation)

texts:
I'm trying to build a model to segment brain tumors.

I trained a model, and the validation dice coefficient is disappointing(0.6). When i saw the predicted images with the ground truths, it seems like the model can't segment tumors well in unusual images like the right of the below. The left image is an usual brain tumor image. (please note the color of the tumor)

And i'm not sure, but the dice score seems to be related with the input range of tumor region.

low dices - 0 centered input
high dices - about 3 centered input.

The input were standardized.

Questions:
1. How does i make the model think of 0 centered input as useful information?
2. Does it worth to try other normalization methods(e.g. min-max scale) in this case?
3. In this case, what methods are helpful to improve the performance?


texts: Looking for a strong Phd Topic in Predictive Analytics in the context of Big Data

texts: I'm going to start a Computer Science phd this year and for that I need a research topic. I am interested in Predictive Analytics in the context of Big Data. I am interested by the area of Education (MOOCs, Online courses...). In that field, what are the unexplored areas that can help me choose a strong topic? Thanks.


texts: from each cell of lstm, what are the output's and what does they signify? i understand that there will be three outputs. A long term memory, short term memory and a output. But, i am little confused from colah blog which can be found here. Here he shows that there will be three outputs, one is long term state and other two outputs are exactly same. What is the use of two outputs being same?


texts: Dimensionality Reduction. How to explain dynamics of feature subset based on all features data?

texts:
I have features: f1..f1000. I want to explain dynamics of particular features subset: f1-f5 based on all features data (based on f1..f1000).

Preferably I think of something like "partial PCA", where some Principal Components created on all features will explain dynamics only of 5 of them (not all of them).

If you know any kind of such Dimensionality Reduction technique I would be very glad for your help. Thanks!


texts: How should the bias be initialized and regularized?

texts:
I've read a couple of papers about kernel initialization and many papers mention that they use L2 regularization of the kernel (often with $\lambda = 0.0001$).

Does anybody do something different than initializing the bias with constant zero and not regularizing it?

Kernel initialization papers

Mishkin and Matas: All you need is a good init
Xavier Glorot and Yoshua Bengio: Understanding the difficulty of training deep feedforward neural networks
He et al: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification


texts: Preprocessing for Text Classification in Transformer Models (BERT variants)

texts:
This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models?
I remember for training a Word2Vec or Glove, we needed to perform an extensive text cleaning like: tokenize, remove stopwords, remove punctuations, stemming or lemmatization and more. However, during last few days I have had a quick jump into transformer models (fascinated btw), and what I have noticed that most of these models have a built-in tokenizer (cool), but none of the demos, examples, or tutorials are performing any of the these text preprocessing steps. You may take fast-bert for instance, there are no text preprocessing involved for the demos (maybe it is just a demo), but at inference the whole sentences are passed without any cleaning:
texts = ['I really love the Netflix original movies',
         'this movie is not worth watching']
predictions = learner.predict_batch(texts)

The same is true for the original transformer by HuggingFace. Or many tutorials that I have looked at (take this or another one). I can imagine that depending on the task this might not be required, e.g. next work prediction or machine translation and more. More importantly I think this is part of the contextual-based approach that these models offer (that is the innovation so to say) that are meant to keep most of the text and we may obtain a minimum but still good representation of the each token (out of vocabulary word). Borrowed from medium article by HuggingFace:

Tokenisation
BERT-Base, uncased uses a vocabulary of 30,522 words. The
processes of tokenisation involves splitting the input text into list
of tokens that are available in the vocabulary. In order to deal with
the words not available in the vocabulary, BERT uses a technique
called BPE based WordPiece tokenisation. In this approach an out of
vocabulary word is progressively split into subwords and the word is
then represented by a group of subwords. Since the subwords are part
of the vocabulary, we have learned representations an context for
these subwords and the context of the word is simply the combination
of the context of the subwords.

But does that hold true for tasks like multi-label text classification? In my use case the text is full of not useful stopwords, punctuation, characters and abbreviations and it is multi-label text classification as mentioned earlier. And in fact the prediction accuracy is not good (after a few rounds of training using fast-bert). What do I miss here?


texts: Question about (Python/Orange) Apriori associative algorithm

texts:
I'm trying to wrap my head around Association rules and frequent itemsets.
So I threw my data in, instead of the samples one and sometime it works, sometimes it doesn't.

rules = Orange.associate.AssociationRulesSparseInducer(data, support = 0.3)

print "%5s   %5s" % ("supp", "conf")
for r in rules:
    print "%5.3f   %5.3f   %s" % (r.support, r.confidence, r)

inducer = Orange.associate.AssociationRulesSparseInducer(support = 0.2, store_examples = True)
itemsets = inducer.get_itemsets(data)
print itemsets
print data.domain
print [data.domain[i].name for i in itemsets[4][0]]

More often than not, itemsets[4][0] shows an IndexError: list index out of range error.
So I start playing around with support = 0.3, support = 0.5, support = 0.2 and itemsets[2][0] or itemsets[3][0].

From the docs:

  support
  Minimal support for the rule. Depending on the data set it should be set to sufficiently high value to avoid running out of working memory (default: 0.3).

True - I tried 0.2 and it quickly blasted my memory on a 800 rows data file.

Any idea what I should do best there or which are viable values for a shopping cart analysis?

800 rows of data (800 orders)
1 to x item(-categories) per order
15 different item-categories in the file, so my data looks like:

ItemCat1
ItemCat2, ItemCat2, ItemCat2, ItemCat2, ItemCat7, ItemCat7, ItemCat7,     ItemCat7, ItemCat7
ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat2
ItemCat4, ItemCat4
ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat1, ItemCat2
ItemCat5


texts:
I have been using LDA to try to build a disease prognosis using medical data for a group of patients and a group of controls, and after limited success I decided to try to use kernel Fisher discriminant analysis. I wrote code in Matlab based on the paper by Mika.

This method seems to be very very sensitive. For example using say 80 rows of data this method is amazingly good at seperating the data into the 2 classes accurately (no overlap at all), but removing the first row of the data and then using this method leads to a terrible discrimination between classes.

Any ideas on why this might be occurring would be greatly appreciated.
I have 24 measurements from each patient and the kernel map I have been using is a polynomial map, with $$k(x,x')=(x \cdot x')^2.$$ This corresponds to mapping $\mathbb{R}^{24}$ into $\mathbb{R}^{576}$, sending $(x_1,x_2,x_3,\dots, x_{24})$ to $(x_1^2,x_1x_2,\dots,x_1x_{24},x_2x_1,x_2^2,\dots, x_{24}^2)$


texts: How to output my uploaded text in Shiny in R at the position of the screenshot?

texts:
I would like to position a long text on the right side of this Shiny app, as shown on the screenshot below.
    library(quanteda)
    library(shiny)
    library(tm)
    library(tidytext)
    library(tidyverse)
    library(shinydashboard)
    library(shinythemes)

    war <- readLines("war.txt")

    war_corpus <- corpus(war)

    sentences <- tokens(war_corpus,what="sentence")

    make_sentences <- function(word) {
      grep(word,sentences,value=TRUE)}

    make_sentences("prince")

    ui<- shinyUI(fluidPage(

  # Application title
  mainPanel(
    img(src='image.jpg', align = "right"),

    #titlePanel(title=div(img(src="image.jpg"))),

    fluidRow(HTML(" Search Bar")),
    #fluidRow(HTML(" Date: 06-29-2020") ),

    fluidRow(
      br(),
      p("User Name : ")),
    br(),
    br(),

    fluidRow(HTML("Enter a word.Click \"Next words\" after") ),
    fluidRow( p("\n") ),

    # Sidebar layout
    sidebarLayout(

      sidebarPanel(
        textInput("inputString","Enter a word here",value = " "),
        submitButton("Next words")
      ),

      mainPanel(
        h4("Are you looking for this ?"),
        tags$style("#mytext {white-space: pre-line;}"),
        verbatimTextOutput("mytext")

      )
    )
  )))

    server <- function(input, output, session) {
      output$mytext <- renderPrint({

        sentences <- make_sentences(input$inputString)
        length(sentences)
        cat(paste0(1:length(sentences),"  ",sentences,sep= '\n'))
      })
    }

here is the rcloud link of the project to experiment with : https://rstudio.cloud/project/1497912


texts: What do this Classification evaluation results mean to you? Do they are suspicious or not?

texts:
I have collected dataset with two class labels and used the SVM Method to classify the dataset, and this is the results. Does this appear suspicious or not?

scikit-learn classifiers with SVM SVC train on 114859 instances, test on 49227 instances
Excution (Training) Time:  9.82799983025
Excution (Testing) Time:  3.75
accuracy: 0.999837487558
Precision-Recall AUC: 1.00
ROC AUC: 1.00
pos precision: 0.999822253822
pos recall: 1.0
pos F-measure: 0.999911119012
neg precision: 1.0
neg recall: 0.998107404779
neg F-measure: 0.999052806062


texts: Is zero-inflated negative binomial regression appropriate for this data? Am I interpreting it correctly?

texts:
I am evaluating whether governance predictor variables are associated with the prevalence of groundwater fecal contamination in a developing country context, as measured by TTC (Thermotolerant Coliform) counts per 100mL of water. In my data TTC is distributed non-normally. There are many zeroes, and also many water sources with TTC of 125+ (our test kits could not measure TTC above this threshold). I ran countfit on TTC and various predictors and it appeared to indicated ZINBR was the appropriate regression to run here.

My questions are:
1) Is zero-inflated negative binomial regression the right regression type to run with this data?
2) When STATA asks me to specify an inflation variable, should I specify ALL predictor variables, or only those that I expect might be driving the over-representation of zeroes in the dependent variable? For example, if my hypothesis is that there is a general association between some governance variable and TTC, but I ALSO think that TTC is affected by the type of water source and that water source type might explain the zeroes (say, because deep, mechanically drilled wells rarely become contaminated regardless of governance while shallow hand dug wells are more susceptible to environmentally-induced contamination), should I only specify water source type as my inflation variable, or should I also specify the governance variables that I am testing as inflation variables?
3) When I interpret the output, am I correct in understanding that an Incident Risk Ratio of 0.74 associated with a binary variable means that if the binary variable is present, predicted TTC = intercept*0.74? (see image 5)
4) When I interpret the output, am I correct in understanding that an Incident Risk Ratio of 0.99 associated with a continuous variable such as percentage of a community with sanitation access, would mean that for each increase of 1% sanitation access, you multiply the intercept*0.99, so that if sanitation access= 10%, your predicted TTC= intercept * 0.99^10? (see image 5)
5) If I include a variable both in the main model as well as the logit model (I'm unsure if that's actually permissible, see question 2), and if the variable is significant in either the main model or the logit model but not both, how am I to interpret that (see image 6)?


texts: ValueError: Numpy arrays that you are passing to your model is not the size the model expected

texts:
I am trying to perform concatenation on the Bidirectinal LSTM layer. I have my model defined like this:

embedding_layer=Embedding(words,300,input_length=trainDataVecs.shape[1],weights=[embedding_matrix])

LSTM_word_1 = Bidirectional(LSTM(80, activation='tanh', dropout_W = 0.25, dropout_U = 0.25,return_sequences=True))
model1 = Input(shape=(trainDataVecs.shape[1],),dtype='int32')
embedded_sequences1=embedding_layer(model1)
lstm_word_out_1 = LSTM_word_1(embedded_sequences1)

LSTM_word_2 = Bidirectional(LSTM(80, activation='tanh', dropout_W = 0.25, dropout_U = 0.25,return_sequences=True))
model2 = Input(shape=(trainDataVecs.shape[1],),dtype='int32')
embedded_sequences2=embedding_layer(model2)
lstm_word_out_2 = LSTM_word_2(embedded_sequences2)

conc=concatenate([MaxPooling1D()(lstm_word_out_1),AveragePooling1D()(lstm_word_out_2)])

outp = Dense(classes_num, activation='sigmoid')(conc)
model=Model(input=[model1,model2],output=outp)

I am fitting like this:

history=model.fit(trainDataVecs,Y_train,shuffle=True,batch_size=512,epochs=10,validation_data=(valDataVecs,valid_Y))

But I am getting the error like:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays
PS: The training and validation vectors are arrays. But still I am getting the error. I think it might be because of the model definition but I am not sure.


texts:
I am using crypto currency chart data and was wondering what would be the best process for scaling the time series data. The issue is that some of these currencies are highly volatile and can see extreme swings. The standard process that I see is something like:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
x_train, y_train, x_test, y_test = train_test_split(x,y, test_size=0.2)
scaler = MinMaxScaler()
scaler.fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)

By default MinMaxScaler scales between 0 and 1 but there are cases where x_test may contain a max that could be double or more what the max was when the scaler was fit. Will this cause model issues or is this acceptable? How do you handle scaling data that you don't know the min/max for? Is min/max scaling best for this?

Also, some currencies have drastically different prices. For example, one could be trading +10x or -10x or more per unit than another. Will having several time series on drastically different scales effect the models? Its seems to me that scaling each time series and each feature to 0-1 would be the best but I am not completely sure. Lets say I have n different time series that have drastically different values. Would something like the following be a decent approach?

#assume data is a list of padas DataFrames
#e.g. data = [pd.DataFrame(asset1), pd.DataFrame(asset2)]
scaled = []
for asset in data:
    fit_data = data.iloc[:int(len(data*0.2))]
    scaler = MinMaxScaler()
    scaler.fit(fit_data)
    scaled.append(scaler.transform(asset))


texts: Cardinality vs width in the ResNext architecture

texts:
I was recently reading the paper Aggregated Residual Transformations for Deep Neural Networks.

One thing the author mentions in Section (5.1) is that increasing the cardinality (or, the number of branches), decreases validation error more than increasing the bottleneck width or increasing the depth. I understand the depth part, but I'm a bit confused about the width. Isn't the cardinality of a residual block the same as the bottleneck width? If not, what is the difference?

Thanks!


texts: Applying activation on part of the layer in Keras

texts:
Context
I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network:
i = Input(shape=(image_height,image_width, image_channels))
rescaled = Rescaling(1./255)(i)
x = Conv2D(16, (1, 1))(rescaled)
x = Conv2D(32, (3, 3))(x)
x = LeakyReLU(alpha=0.3)(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Conv2D(16, (3, 3))(x)
x = Conv2D(32, (3, 3))(x)
x = LeakyReLU(alpha=0.3)(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Flatten()(x)
x = Dense(256, activation='sigmoid')(x)
x = Dense(grid_width * grid_height * anchor_number * (5 + class_count))(x)
x = Reshape((grid_width, sgrid_height, anchor_number, (5 + class_count)))(x)

Which supposed to output, for each grid cell and anchor box, a vector of $(p(c), b_x, b_y, b_h, b_w, class_0, class_1, ..., class_n))$
The problem
Some of the output vectors, namely $p(c)$, $b_x$ and $b_y$, are limited to be between 0 and 1, so they should pass through a sigmoid activation. The part of $class_0, class_1, ..., class_n$ is a classification, so it should pass through a SoftMax activation. So I need a way to specify what part of the output needs to use which activation.
TL;DR: How do I apply different activation functions to different parts of the network output?


texts:
When we fit an LSTM model, each LSTM has a cell state which contains the information we want. I am wondering what's the dimension in the cell state (i.e. cell state itself should be a vector, then how large is the vector?)?

(from http://colah.github.io/posts/2015-08-Understanding-LSTMs/ )

i.e. In the above architecture, how do we determine the dimension of the matrix Wc?

If I am using Keras package, how do I find the Cell state dimensions of each LSTM unit? Is it something that one can adjust?

Thank you!


texts: Siamese Neural Network for massive class variation

texts:
I have case to check whether the person has been registered into a database,and if the images has high similiarity with one of the image in the db, i want to retrieve the image information (name, id, etc) to email or other party. The problem is the dataset are very limited (I only get 1 image per person) so i augmented the image and get about 30 variation of augmented faces. At first i have thinking using Siamese NN since it has good performance with limited data source, but realise its need pair of image (the gaithered and one from the dataset), loop through big dataset (need to loop through 3k of image), SNN is not too efficient. I afraid using VGG or Facenet since the augmented dataset are not too big enough to gain certain accuracy.
Is there any suggestion based on my case? The one idea i have to tackle it is to use NN as feature and use similarity calculation based from stored NN.


texts:
My training data is weighed heavier on the '1' class, with about a 4:6 ratio. This outputs a classifier that is of 82% accuracy with an emphasis on the '1' class, which makes sense.

Confusion Matrix -
[[333 133]
 [ 62 612]]

I have the test proportions as well, in which the data will be tested on, which is 0.3 of '1' and 0.7 of '0' or 1900 0s and 900 1s. My classifier outputs 1400 1s and 1300 0s.

My theory is that I need to build a classifier that favours the '0', If so how can I make the classifier biased to one class over another?

I have tried to used the class weights, this does increase the '0' predictions but only by a very small percentage.


texts: Orange Datamining don't plot lines along the dates

texts:
Why Orange don't plot the line along the dates from the following table?

I have two columns, confirmed (numeric) and data (time).
But plot line doesn't appear along the dates. It seems to me that it tried to plot both columns.


texts: Overlaying a line graph and an area graph: adding recession bars to a time series

texts:
I am trying to reconstruct a time series graphs from FRED: https://fred.stlouisfed.org/series/LABSHPUSA156NRUG using Python. However, I am unable to get a clean figure where the time series is overlaid with the area graph representing the recession bars.
I am quite new to python so it's been frustrating to find a simple reason why my attempts are not working.
I have attempted this two ways.
My first attempt looks like this:
fig, ax = plt.subplots()
ls_data.plot.line(ax=ax, figsize=(8,5), x='Date', color=blue)
rec_data.plot.area(ax=ax, figsize=(8,5), x='Date', alpha=0.5, color=gray)
plt.ylim(0.59,0.65)

And produces the following perplexing mess:
My second attempt goes for the "ax2 = ax1.twinx()" route
fig, ax1 = plt.subplots()
ax2 = ax1.twinx()

ls_data.plot.line(figsize=(8,5), x='Date', color=blue)
rec_data.plot.area(figsize=(8,5), x='Date', alpha=0.5, color=gray)

However this produces two separate figures instead.

One possible issue is that the x-values for the labor share are plotted in individual years whereas the x-values for recession are plotted in months. But based on the second attempt, it looks like the graphs should overlay smoothly but then I'm liable to get the same result as in my first attempt. Would transforming the recession data provide a fix?
Edit: The data for recession dates are available here. https://fred.stlouisfed.org/series/USREC#:~:text=For%20daily%20data%2C%20the%20recession,the%20month%20of%20the%20trough.


texts:
I'm trying to visualize the neural network architecture.

from keras.utils import plot_model
plot_model(model, to_file='model.png', show_shapes=True)

What does the huge number on top mean?


texts: Predictions with arbitrairy sequence length for stateful RNN (LSTM/GRU) in Keras

texts:
I have time series data of the following properties:

input shape:  (num_timesteps, num_features)
output shape: (num_timesteps, num_outputs)

I reshape it to batch form:

input shape:  (num_batches, num_timesteps_in_batch, num features)
output shape: (num_batches, num_timesteps_in_batch, num outputs)

I have a stateful RNN in Keras:

modelinput = Input(batch_shape=(num_batches,None,num_features))
prediction = GRU(10,return_sequences=True,stateful=True)(inputs)
model = Model(inputs=modelinput,outputs=prediction)

After trainig (which works fine) I would like to predict on a sequence without cutting the data, so input shape (num_timesteps, num_features). How can I do that?

I thought about having a second model that shares the weights with the RNN and that has dynamic input shapes. Is that possible?


texts: What's the best way to tune the regularization parameter in neural nets

texts:
I'm tuning the regularization parameter of a neural net (L2 regularization) using a grid. Starting with values 0.0005, 0.005, 0.05, 0.5, 5. Then if 0.005 brigs the best validation result, I continue using a grid like this: 0.0015, 0.0025, 0.0035, 0.0045, 0.0055, 0.0065, 0.0075, 0.0085, 0.0095. And it continues..

I would like to know if there's a more intelligent way to tune this parameter.


texts: wrong output for neural network written from scratch

texts:
I am trying to write a complete neural network from scratch. However, the results are not very encouraging. My prediction output shows one for one single row only in the 3 class classifier. I will post my code and then I will talk about the problem in the output.

#labels or classes
#1=iris-setosa
#2=iris-versicolor
#0=iris-virginica

#features
#sepallength
#sepalwidth
#petallengthcm
#petalwidth

import pandas as pd
import matplotlib.pyplot as plt
import csv
import numpy as np
df=pd.read_csv('Iris.csv')

df.convert_objects(convert_numeric=True)
df.fillna(0,inplace=True)

df.drop(['Id'],1,inplace=True)
#function to convert three labels into values 0,1,2
def handle_non_numericaldata(df):
    columns=df.columns.values
    for column in columns:
        text_digit_vals={}
        def convert_to_int(val):
            return text_digit_vals[val]
        if df[column].dtype!=np.int64 and df[column].dtype!=np.float:
            column_contents=df[column].values.tolist()
            unique_elements=set(column_contents)
            x=0
            for unique in unique_elements:
                if unique not in text_digit_vals:
                    text_digit_vals[unique]=x
                    x+=1
            df[column]=list(map(convert_to_int,df[column]))
    return(df)
handle_non_numericaldata(df)

x=np.array(df.drop(['Species'],1).astype(float))
c=np.array(df['Species'])

n_values=(np.max(c)+1)
#to make the vector one hot encode
y=(np.eye(n_values)[c])
(np.shape((y)))
m1=np.size(c)

theta=np.ones(shape=(4,21))
theta2=np.ones(shape=(21,3))
#no of examples "m"

#learning rate alpha
alpha=0.05
#regularization parameter
lamda=0.05
for i in range(1,5000):
    z1=np.dot(x,theta)
    sigma=1/(1+np.exp(-z1))

    #activation layer 2.
    a2=sigma
    z2=np.dot(a2,theta2)

    probs=np.exp(z2)

    softmax=probs/np.sum(probs,axis=1,keepdims=True)
    delta3=softmax

    delta3=delta3-y
    A2=np.transpose(a2)
    dw2 = (A2).dot(delta3)
    W2=np.transpose(theta2)

    delta2=delta3.dot(W2)*sigma*(1-sigma)
    X2=np.transpose(x)
    dw1=np.dot(X2,delta2)

    dw2=dw2-lamda*theta2
    dw1=dw1-lamda*theta

    theta =theta -alpha* dw1

    theta2= theta2-alpha * dw2

final1=x.dot(theta)
sigma=1/(1+np.exp(-final1))
z2=sigma.dot(theta2)
exp_scores=np.exp(z2)
probs=exp_scores/np.sum(exp_scores,axis=1,keepdims=True)
a=np.zeros_like(probs)
a[np.arange(len(a)),a.argmax(1)]=1
print(a)

The output according to me, as I have written this, all the output probabilities are same hence  it cannot classify properly.output should be 'a' array with 50 1's in the first row, 50 in the second, and 50 in the third. But all the ones are in first row. I have been meddling with this since last three days. Please help me figure out mistake I have made in writing this neural network.


texts: How many normalization methods are there and what are they for?

texts:
this post lists 5 types of normalization.

Zscore
MinMax
Logistic
LogNormal
TanH

is there any other types of normalization that are most commonly used in machine learning?


texts: Should I normalise image pixel wise for pretrained VGG16 model

texts: My goal is to use pretrained VGG16 to compute the feature vectors excluding the top layer. I want to compute embedding(no training involved) per image one by one rather than feeding batches to the network as this network is just to compute embedding and not classification. In batch training I understand the importance of batch normalisation but for a single image should I normalise it pixel wise? All I can think is maybe its useful to reduce the importance of illumination in an image. Am I missing something?


texts: logistic regression algorithm fails to work

texts:
I'm trying to code my own logistic regression algorithm using Andrew NG's machine learning using Octave. lectures. So what I did was make a csv file, the first row being some parameter and the second one being the result:

121,1
124,0
97,0
104,0
110,0
...

Overall there are only 24 examples, but I've chosen points such that some pattern can be followed.

Here is my code:

data = load('data.dat');
x = data(:, 1);

y = data(:, 2)
m = length(y);

#plot(x, y, 'rx', 'MarkerSize', 10);
#xlabel('IQ');
#ylabel('Pass/Fail');
#title('Logistic Regression');

x = [ones(size(x, 1), 1) x];
alpha = 0.00001;
i = 15000;

g = inline("1 ./ (1 + exp(-z))")

theta = zeros(size(x(1, :)))';
j = zeros(i, 1);

for num = 1:i
  z = x * theta;
  h = g(z);
  j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))
  grad = 1./m * x' * (h - y);
  theta = theta - alpha * grad;
end

However the output of the sigmoid function shows every value below 0.5... surely this has to be wrong. I've also tried with different learning rates and iterations, but to no avail. What is wrong with the code, or data?

Help would be appreciated.


texts: Predicting products to be sold in a store - problem formulation

texts:
I have a data from a store for the products that sold since more than 5 years. Each sell process has a customer id, date, and the quantity of the product.
I want to build a machine learning model to predict the products that will be sold in the next day/s for each of the customers, giving that I have N products (~2k) and M customers (~50).
I am not able to formulate this problem. It's a regression task (probably), but I don't know how can I formulate it to predict the products that a given customer will buy.
Since we have N products, this doesn't mean that a customer will buy all of them; x customer might buy only 5 products in the next day.


texts: How to predict an outcome within a specific time window?

texts:
I have a dataset which has around 10K records.

My objective is to predict whether the customer will churn or not. Binary classification problem with each class representing around 55:45 proportion and 20 features.

I understand when it's just about prediction, I can apply some binary classification algorithm and find out whether the customer churns or not

But how do I incorporate the objective of finding whether the customer will churn in 30 days or not?

Another example is find whether patient will be dead within 30 days from the date of discharge. I have his date of discharge along with other features like Blood pressure, Cholesterol etc.

Rather than just predicting whether he will be dead or not anytime in future, I would like to restrict it to 30 days from date of discharge.

Hope I gave the details to help you understand the question better.


texts:
A very important aspect in deep learning is the learning rate. Can someone tell me, how to initialize the lr and how to choose the decaying rate. I'm sure there are valuable pointers that some experienced people in the community can share with others. I've noticed that many choose to do a custom scheduler rather than use available ones.

Can someone tell me why and what influences the change in the lr? And when to describe a lr as being small, medium or large? I want to understand it enough to actually make sound choices. Thank you kind souls. I appreciate this community very much.


texts: Pearson Product Moment Correlation vs Cosine Similarity For Encoded Text Comparison

texts:
I've seen a few different examples of the implementation of Google's Sentence Encoders. Many of these use different methods to find the similarity between sentences.
For example, the standard Universal Sentence Encoder notebook implements np.inner() which calculates the Pearson Product Moment Correlation if I understand properly (please correct me if I'm wrong). It normalizes the vector first and then performs the dot product. https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb
The Universal Sentence Encoder notebook implements, however, implements cosine similarity when comparing how similar two words/sentences are. https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/cross_lingual_similarity_with_tf_hub_multilingual_universal_encoder.ipynb
In the couple examples I've tried so far, the np.inner() method seems to be more effective although this is very anecdotal. Specifically, I'd like to get a better idea of what the difference between normalized dot product and cosine similarity is and when each should be used.


texts:
There is a function: $f(t,x,y,z)$, where $t$- time, $x,y,z$- some arguments.

The values of $f$ for $t\in [a,b]$ are known (100 samples). What is the most accurate way of predicting the value of $f$ for $t=b+1$, if the values of $x,y,z$ for $t\in [a,b+1]$ are known?

May I implement deep-learning, and if yes, which method would be more reliable?

ADDITIONAL INFO

I'm propagating the satellite position using an analytical method. During the propagation, the errors accumulate. I endeavor to predict the future position errors. $f$ represents the error value and arguments $x,y,z$ are the affecting parameters, such as distance to the Moon, Sun, solar flux, atmospheric density, etc.


texts: using sklearn class weight to increase number of positive guesses in extremely unbalanced data set?

texts:
Hi I have a poorly correlated and unbalanced data set I have to work with. The set is 2 classes, 0 has 96,000 values and 1 has about 200. When I run random forest or other methods I get an output like:

    precision    recall  f1-score   support

          0       1.00      1.00      1.00     38300
          1       1.00      0.01      0.02        90

avg / total       1.00      1.00      1.00     38390

Precision is very high but it only classified one row as positive?

I tried using {class_weight = 'balanced'} in the random forest parameters and it provides:

   micro avg       1.00      1.00      1.00     38390
   macro avg       1.00      0.51      0.51     38390
weighted avg       1.00      1.00      1.00     38390

But still not many positive guesses? Should I look into oversampling?


texts:
From what I have understood about model stacking: the meta estimator trains to combine the N-models predictions to fit the ground truth. Once trained, it combines the 1st level output to approach the ground truth.

The meta estimator is a model of type :
$ (y_{pred1}, y_{pred2}, y_{pred3})\rightarrow y_{pred-stack}$

So the combination is only based on the values of 1st level predictions. However, each line of the stacking data is also linked to other attributes: "Brand", "Model", "Power". Why won't we take those attributes to determine the optimal combination? So if the model 1 is the best when the brand is "NaN", the meta will learn it and redirect every prediction having NaN brand to model 1.

So the meta estimator I propose is as follow :
$ (y_{pred1},y_{pred2},y_{pred3},$brandIsNull$)\rightarrow y_{pred-stack}$

Does this approach exist?
If not, would it be a good or bad idea?


texts: How to interpret the sample_weight parameter in MiniBatchKMeans?

texts:
I am using scikit-learn MiniBatchKMeans to do text clustering.
In the fit() function there is a parameter sample_weight described as follows:

The weights for each observation in X. If None, all observations are
assigned equal weight (default: None).

How should I interpret it exactly?
For instance, if I assigned greater weight to certain observations, will the centroid be nearer to that observations?


texts: In some implementations of SRGAN I've noticed, that datasets consist of the high-resolution images and the low-resolution images are created later by, e.g. resizing (decreasing the size) hr-images. However, e.g. I have a dataset of both hr-images and lr-images, all of them are of the same size but lr-images are of lower quality (less details, blurred). Is it proper to use SRGAN in such case or should I use another way of image enhancing?


texts: How to convert nested list into a single list in python?

texts:
I have a list that contain id number. Some elements of the list is an another list. To convert nested list into a single list, I write a recursive function using collections module.

My codes are following.

from collections import Iterable
def single_list(list):
for item in list:
    if isinstance(item, Iterable):
        yield from single_list(item)
    else:
        yield item

Item_list = [10,20,[30,40],[50,'Null',70],100]
items_single=single_list(Item_list)
for item in items_single:
print(item)

When I run my program, I get the following error message.

  Traceback (most recent call last):
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 42, in <module>
  for i in items_single:
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 36, in single_list
  yield from single_list(item)
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 36, in single_list
  yield from single_list(item)
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 36, in single_list
  yield from single_list(item)
  [Previous line repeated 986 more times]
  File "/Research/SoftDev/SEPJ/StackOverflow_qs.py", line 35, in single_list
  if isinstance(item, Iterable):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/abc.py", line 184, in __instancecheck__
if subclass in cls._abc_cache:
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_weakrefset.py", line 75, in __contains__
return wr in self.data
 RecursionError: maximum recursion depth exceeded in comparison

 Process finished with exit code 1

I do not know how to fix the error.


texts: Convolutional neural network for sparse one-hot representation

texts:
I have some basic features which I encoded in a one-hot vector.

Length of the feature vector equals to 400.
It is sparse.

I saw that conv nets is applied to a dense feature vectors.

Is there any problems to apply conv nets to a sparse feature vectors?


texts:
I would like to ask a question about recommender systems. We are showing some movies to users and they have to decide if they like them or not. These movies have only a few attributes

Title
Director
Category
Duration
Nationality

We show the attributes as a block. The users can decide if they like (the block) or not. We aggregate these decisions to recommend them more movies in the future. My question is: Are there any methods that can tell us whether a user is uninterested in a given attribute? For example, if a user is never considering duration when saying yes or no.


texts: Can multiple time series take advantage of each other?

texts:
I want to forecast house prices market in multiple cities of the same country. I expect demography, interest rate and neighbors cities values to have an impact on my prediction. For every city, I have a limited amount of data points (+-60). I use Python and I am open to any library.
I would like not only to rely on the prices historical data but also on other independant variables (population growth, ...). I found that Transfer Functions Models could be the way to use other time series to forecast the prices.
I have 1 model for every city so 500 in total. Each model will use population to price relation to improve prediction. Is there a way to combine the relations found in each of them to improve global performance ?
There are 30.000 data points in total (60 * 500 cities) so it is pity that each city's model look at its small subset because the relations between population growth and prices increase are probably similar accross all cities.

Is there a way to create a big model (without neural network) to take advantage of all points, but can predict on each city independently ?
Is there a way to create connections between the 500 models, so they can benefit of the learning of their peers ?


texts: How is the concept of data different for different disciplines? Obviously, for physicists and sociologists, "data" is something different.


texts:
I have an hypothesis but i don't know if it's true.

If the cluster is dense and we apply a supervised learning on this data, the model generated by this cluster will be more efficient for new data falling into this cluster than other.
Thus we have multiple cluster with associate models

How choose the right associate model to optimize our prediction ?
We can compare the value of the point to cluster's mean and apply the right model !

Any suggestions ?


texts: What does "Model recursive loss convergence" mean?

texts:
I trained a very simple neural network to figure out some features from an image. it's something like a reverse graphic. The neural Network will receive an image and try to estimate its features. The features can completely describe the image(like inverse graphic-see p.s.2). So we can use them to reconstruct the image using a "rendering function".
Obviously, the network has some error and the image cannot be reconstructed exactly.
I fed the 'Reconstructed Image' to the model and got some new 'Estimated Features'. I repeated this and after some time the 'Estimated Features' has converged to some value. Not the 'True Value' for sure but something.
What does this repeated estimation convergence and also the loss convergence of this repeated process mean? can it be useful for anything? for example, for training the Neural Network model in a more efficient way.
This is the plot of the convergence of the estimated parameters and the real value of each of them. Colors represent each Param. lines are estimated values in each feedback loop iteration and '+' is the real value.

p.s.1: the loss is too high cause I wanted to show how it converges to something. with more training, the Loss decrease and the distance get's really small.
p.s.2: For the reverse graphic, you can imagine a set of features like (x, y, width, height, thickness) which can be used to draw a rectangle. The rendered image then can feed into a model to find the estimated features of (_x, _y, _width, _height, _thickness)
p.s.3: The 5th feature's True and estimated value is 0 for all samples


texts: How to deal with situation where LSTM fails to learn (constantly makes the same incorrect prediction)

texts:
I am trying to use LSTM neural networks in order to make a song composer. Basically this is based of a text generator (tries to predict the next character after looking at a sequence of characters) but instead of characters, it tried to predict notes.

Structure of the midi file that serves as the input (Y-axis is the pitch or note value while X-axis is time):

And this is the predicted note values:

I set an epoch of 50, but it seems that the LSTM's loss rate does not decrease, most of the time its loss rate does not improve.

I suspect this is because there is an overwhelming number of a particular note (in this case, note value 65) which makes the LSTM lazy during training phase and predict 65 each and every time.

I feel like this is a common problem among LSTMs and time-series based learning algorithms. How would I solve a problem like this? If what I mentioned is not the problem, then what is the problem and how do I solve that?