texts: Vanishing problem with cyclegan wasserstein loss function

texts:
I have modified a keras cyclegan keras cyclegan version of horses and zebras to the classical fer2013 face recognition file.
I got some results this cyclegan trying to get some additional DISGUST (disgust is the case with fewer samples in the file) faces from NORMALS and I'm trying to modify the discriminator loss with wasserstein as stated here:
But after some epochs I see that after every epoch result face is the same no matter the input face is fed. I conclude that this is the famous vanishing problem.
This is the colab network. You can see the images that result in the last epochs...Any help would be appreciated in order to prevent this to happen.

This is the code of loss function (from colab):

# implementation of wasserstein loss
def wasserstein_loss_fn(y_true, y_pred):
    return K.mean(y_true * y_pred)

# Define the loss function for the generators
def generator_loss_fn(fake):
    fake_loss = wasserstein_loss_fn(tf.ones_like(fake), fake)
    return fake_loss

# Define the loss function for the discriminators
def discriminator_loss_fn(real, fake):

    # Add random noise to the labels - important trick!
    real_labels = tf.ones_like(real)
    real_labels += 0.05 * tf.random.uniform(tf.shape(real_labels))
    fake_labels = tf.ones_like(fake)
    fake_labels += 0.05 * tf.random.uniform(tf.shape(fake_labels))

    real_loss = wasserstein_loss_fn(real_labels, real)
    fake_loss = wasserstein_loss_fn(fake_labels, fake)
    return (real_loss - fake_loss) * 0.5


texts: Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy?

texts:
My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima".
In my opinion this does not work at all, and should lead to picking the model that best fits the test_set, but may not be generalizable enough for it to work with the actual distribution behind it.
Is there any material or research on this? I really think this method is archaic and makes no sense, but i can't argue with my professor without actual scientific material.


texts: what is the interest of TimeDistributed after an LSTM layer?

texts:
I've already seen several similair questions but I did not understand anything,
what is the interest of TimeDistributed?
why we need to insert a TimeDistributed layer after LSTM to establish the time series prediction


texts: Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another

texts:
I have a very basic question which relates to Python, numpy and multiplication of matrices in the setting of logistic regression.
First, let me apologise for not using math notation.
I am confused about the use of matrix dot multiplication versus element wise pultiplication. The cost function is given by:
$J = - {1\over m} \sum_{i=1}^m y^{(i)}log(a^{(i)})+(1 - y^{(i)})log(1-a^{(i)})$
And in python I have written this as
    cost = -1/m * np.sum(Y * np.log(A) + (1-Y) * (np.log(1-A)))

But for example this expression (the first one - the derivative of J with respect to w)
${\partial J \over{\partial w}} = {1 \over{m}} X(A-Y)^T$
${\partial J\over{\partial b}} = {1\over{m}} \sum \limits_{i = 1}^m (a^{(i)}-y^{(i)})$
is
   dw = 1/m * np.dot(X, dz.T)

I don't understand why it is correct to use dot multiplication in the above, but use element wise multiplication in the cost function i.e why not:
   cost = -1/m * np.sum(np.dot(Y,np.log(A)) + np.dot(1-Y, np.log(1-A)))

I fully get that this is not elaborately explained but I am guessing that the question is so simple that anyone with even basic logistic regression experience will understand my problem.


texts: Document classification - optimal classifier & embedding

texts:
Planning a classifier to recognize specific documents based on text. Categories are mutually exclusive and include: job application resumes, paychecks, quarterly financial reports etc, and data is text extracted from doc, pdf, xls etc.

At the feature engineering stage, with questions in mind:

One Multiclass classifier or a number of One vs. Rest (one per category)?
Word embedding (Word2Vec/GloVe) - one embedding for all categories or a separate embedding for each category?

My tendency is towards a number of one vs. rest classifiers (to allow for dynamic use of just some of the categories), and one embedding for all categories (to save prediction time).

Project architecture: initially basic LogReg, but will soon develop to CNN / CNN+RNN.

Project scale:

Training: a few hundreds per category
Prediction: millions of files

Please feel free to share any insights or references, thank you


texts:
I have the following code
rf = RandomForestClassifier()
rf.fit(X_train, Y_train)
print("Features sorted by their score:")
print(sorted(zip(map(lambda x: round(x, 2), rf.feature_importances_), X_train), reverse=True))

and I get the following error:
> TypeError Traceback (most recent call last)
>
> ipython-input-109-c48c3ffd74e2> in <module>()
>
> 2 rf.fit(X_train, Y_train)
>
> 3 print ("Features sorted by their score:")
>
> ----> 4 print (sorted(zip(map(lambda x: round(x, 2),
> rf.feature_importances_), X_train), reverse=True))
>
> TypeError: '<' not supported between instances of 'int' and 'str'

I am not sure what I am doing wrong. I only have int and float in my dataframe.


texts: Problem faced when collect data randomly from cluster

texts:
I have a semi structured data set. I need to collect some data (unlabeled) randomly for labeling.

As initiative at first I separated labeled and unlabeled data. Then I convert those data from string to numeric as all data were string.

After that I cluster unlabeled data into 2(cluster 0 and cluster 1) by applying k-means clustering algorithm.

Now when I collect some data randomly from both cluster it takes indexes from both labeled and unlabeled data. But my expectation was, indexes will come just from unlabeled data as cluster was created by using unlabeled data.

I couldn't figured out the problem. I am giving my code below.

train_label.head()

The indexes of labeled data stats from 0 to 343 (344 data) and unlabeled data starts from 344 to 863 (519 data)

train_unlabeled.head()

# define X and y
feature_cols = ['TopLeft', 'TopMiddle', 'TopRight', 'MiddleLeft','MiddleMiddle', 'MiddleRight', 'BottomLeft', 'BottomMiddle', 'BottomRight']

# X is a matrix, hence we use [] to access the features we want in feature_cols
X =train_unlabeled[feature_cols]`

# y is a vector, hence we use dot to access 'label'
y =train_unlabeled['Class']

X_scale = scale(X)
reduced_data = PCA(n_components=None).fit_transform(X_scale)
kmeans = KMeans(init='k-means++', n_clusters=2, n_init=10)
model=kmeans.fit(reduced_data)
reduced_data.shape

labels=kmeans.labels_

Now I was trying to collect 95% data from each cluster (cluster was created by train_unlabeld data)

x = 0.95
i=0
C_i = np.where(labels == i)[0].tolist()
n_i = len(C_i)
sample_i = np.random.choice(C_i, int(x * n_i))
print (i, sample_i)
list1=(sample_i)
x = 0.95
i=1
C_i = np.where(labels == i)[0].tolist()
n_i = len(C_i)
sample_i = np.random.choice(C_i, int(x * n_i))
print (i, sample_i)
list2=(sample_i)

When I print the index while collecting data randomly depending on cluster(0 and 1) I found these indexes

As the index of unlabeled data is started from 344, so how the index of labeled data comes in cluster 0 and ? I didn't understand it.


texts: Compare Classification Performance in Datasets drawn from Different Populations

texts:
I've read some classics about comparison of ML Algorithms i.e.

Dietterich, T. G. (1997). Statistical Tests for Comparing Supervised Classication Learning Algorithms 1 Introduction. Science, 10(7), 1–24. Retrieved from http://dx.doi.org/10.1162/089976698300017197

However I feel totally lost about a specific problem.
Backgrond / Status Quo
I have two dataset ($N_1=552$, $N_2=543$) drawn from different populations.
 Both contain the same set of features and the same criterion (7 class labels).To simplify I will spare the details on preprocessing and hyperparameter tuning.
In the end I have two trained algorithms (i.e. two RandomForests:  $RF_1$ & $RF_2$) for both datasets ($df_1$ & $df_2$) respectively
Goal / Aim
I want to know if it is better to train the algorithm using data drawn from population 1 and evaluate it using data drawn from population 2, or if the opposite is true. So which population generalizes better to the respective other. To be more precise if a measure of the classification performance (i.e. Accuracy or Kappa) for the $RF_1$ (Random Forest trained in dataset 1) tested in $df_2$ is significantly higher (not caused by chance) than the performance for the $RF_2$ (Random Forest trained in dataset 2) tested in $df_1$.

$Acc(RF_1->df_2) >  Acc(RF_2->df_1)$

Question
Is there an apropriate test for that? Is it as simple as the $\chi^2$ or a exact binominal test?

Edit
Or am I comparing apples and oranges, and there is no way one could compare this two classification results? I am very thankful for any direction you can give me.


texts: Neural Recommendation System - Explanation

texts:
Hello I am working on a recommendation problem in which I want to recommend the next best product to a customer. I am using a collaborative filtering approach but I would like to have as a result, the products that most influenced the recommendation. Something like " We are recommending product A because you bought B C and D".
I was doing it with implicit library but I am now exploring other approaches such as NRS.
Does anyone know if it is possible with Keras (or other frameworks) for example?
Thanks


texts:
After searching quite some time for it on Google I could not find a sufficient software/toolbox that can manage trainings of neural networks. I thought of a program that combines visualization techniques without the need to write code as well as having the possibility to compare several trainings of neural networks and be able to store them easily.
Does a program like this exist?

Regards
Lukas


texts: How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix?

texts:
How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.

I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.

This is a question from Neural Network Design (2nd Edition) book by
Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .

Resource : E7.5 p 224-225


texts: How to Normalize image intensities of CT and MRI images (single channel)

texts:
I have data-set which contains MRI and CT images and all of them are labeled.
I want to create MRI-CT classifier. But intensity range of MRI and CT are different. CT ranges between (-1024 and 2000) but MRI ranges between (0 and 1000). Without normalization accuracy of my classifier was 100% both in training and test set. I guess model focused on intensity ranges.
Now my question is how should I normalize them?


texts: Will data science develop into a separate academic field?

texts: Right now data science is often regarded as an interdisciplinary subfield of statistics and computer science. What would it take for data science to develop into a separate academic field (with its own associations, journals, and conferences...)?


texts:
I want to be able to create a model that would be able to classify an image that has been split into 9 parts and jumbled around.

I did see a paper on it but it is quite old (7-8 years old). Could anyone point me towards any resources? Is building a CNN the best approach?

Any help is appreciated.


texts: Identify Bad Products from given parameters using neural networks

texts:
I have a problem at hand to identify Good/Bad Products using given parameters. The number of parameters are in the order of 5000s and there are multiple values for the parameters. However I do not have a labelled set of data which says these are the products that are good or bad.

For Example, Say the parameters are AX, AY, AZ, B, C, DX, DY, etc.
Each of them has a different range. Is decision trees the right approach?

Can classification be applied to this problem?


texts: Simple question related to an ANN diagram showing all of the matrices' dimensions: inputs, weights and output

texts:
I've been reading a few Neural Networks articles for the past week and one thing that I am still trying to grasp is the dimensioning of the matrices on an ANN training. I have created a diagram (based on the example given by the same article) to try to figure it out.  I have inserted the weights related to the neurons that are painted on the diagram.

I am hoping that someone can validate my reasoning, as it is one of my pain points right now.

I appreciate any help.

PS: Was this post supposed to be created on this forum or on Cross Validated? I can see that this topic appears on both.


texts: Can the dependency between variables be deduced from data? And if so, how?

texts:
I have a data set $X$ that consists of $m$ vectors $\vec{x}$ of $n$ real-valued components. Each vector component lies within a corresponding predefined interval of valid values, which is the same for all vectors in $X$. The assumption is that there exists a dependency graph between the components of each vector, which is also the same for all vectors; for example, the value of the component $x_k$ (maybe) depends on the values of both components $x_p$ and $x_q$ for all $\vec{x} \in X$ and $k \neq p \neq q$. However, we do not know the exact structure of this graph. In other words, we suppose that there exists a dependency between the variables of each data point, but we don't know how it is structured. So the problem is to deduce this dependency graph using only the data available in $X$.
My question is: is there a method or algorithm that allows me to solve this problem? And if so, can somebody point me to a source where I can learn more about it?


texts: How to correctly apply the same data transformation , used on the training dataset , on real data in a webservice?

texts:
Let's say I used minmaxscaler while creating my model.
Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?


texts: supervised learning : approximating a priority function

texts:
I am facing the following supervised learning problem:

An object is fully characterized by its position in $R^n$.

There are $m$ objects. There are fully observable (i.e. their positions are always known).

At each time step $t$, exactly one of these object is activated. Activation is fully observable, i.e. the index $a_t$ ($a_t \in [1,m]$) of the object activated at time $t$ is known.

We know that under the hood, activation works this way:  there is a priority function $f$ ($f : R^n \to R$ ) which computes for each time step the priority score of each object. The object for which the priority score was the highest is activated.

The goal is to find (the approximation of) one of the possible priority function that would match a given data-set. A data-set is of size $(m*n+1)*t$ ($m$ positions of dimension $n$, plus the index of the activated object, over $t$ time steps).

As an example, if it turns out there is a hidden fixed beacon, and at each time step $t$ the object the closest to the beacon is activated, then a possible function would be $f(o_{it})=1/d_{it}$ where $d_{it}$ is the distance between the beacon and the object $o_i$ at time $t$.

(if several objects have the same highest priority score, then only one of them is activated, selected randomly).

The function found by the algorithm may be parametric and encoded by a neural network, if this is applicable.


texts: In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

texts:
While reviewing the Transformer architecture, I realized something I didn't expect, which is that :

the positional encoding is summed to the word embeddings
rather than concatenated to it.

  http://jalammar.github.io/images/t/transformer_positional_encoding_example.png

Based on the graphs I have seen wrt what the encoding looks like, that means that :

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot,
while there is also a large amount of positions in the embedding that are only slightly affected by the positional encoding (when you move further towards the end).

  https://www.tensorflow.org/beta/tutorials/text/transformer_files/output_1kLCla68EloE_1.png

So, why not instead have smaller word embeddings (reduce memory usage) and a smaller positional encoding retaining only the most important bits of the encoding, and instead of summing the positional encoding of words keep it concatenated to word embeddings?


texts:
In Chapter 6 (Deep Forward Networks) on Page 193 of Deep Learning they talk about the design of Hidden Units.
The Radial Basis Function is introduced as follows:
$$
h_i = exp\big{(}-\frac{1}{\sigma_i^2}||W_{:,i} − x||^2\big{)}
$$
What does the colon as Index for $W$ mean?


texts:
I trained and saved a model that should predict a sons hight based on his fathers height.
I then saved the model to Pickle.
I can now load the model and want to use it but unfortunately a second variable is demanded from me (besides the height of the father) I think I did something wrong when training the model?

I will post the part of the code wher I think the error is in, please ask if you need more.

#Spliting the data into test and train data
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)

#Doing a linear regression
lm=LinearRegression()
lm.fit(X_train,y_train)

#testing the model with an example value
TestValue = 65.0
filename = 'Father_Son_Height_Model.pckl'
loaded_model = pickle.load(open(filename, 'rb'))
result = loaded_model.predict(TestValue)
print(result)

The error message says:

ValueError: Expected 2D array, got scalar array instead:

array=65.0.

Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Thanks alot in advance.


texts: Cosine similarity between two folders (1 and 2) with documents, and find the most relevant set of documents (in folder 2) for each doc (in folder 2)

texts:
I have one folder named iir, it has 500 txt files. I have another json file named video (with dictionary structure).

I wish to compute: for each of the 500 txt files, find the cosine similarity with all the videos. After this rank the videos for each txt files and save that ranking list in another file.

So far, I am able to read all the txt files. This is the code:

import gensim
import glob
# go to directory
all_files = glob.glob("ExtratingConceptFromVideoScripts/iir/*.txt")
# go to files
indata = tuple()
for filename in all_files:
    with open(filename,'r')as file:
        for line in file.readlines():
            d = line.strip().split( ',' )
            indata = indata +  (d[0], )
print(indata[0])  # print the contents of first file

This is the output:

Online edition (c)2009 Cambridge UP An Introduction to Information Retrieval Draft of April 1

Now, I read the contents of the json file:

import gensim
import glob
# go to directory
all_files = glob.glob('ExtratingConceptFromVideoScripts/data/corpus.json')
# go to files
vid = tuple()
for filename in all_files:
    with open(filename,'r')as file:
        for line in file.readlines():
            d = line.strip().split( ',' )
            vid = vid +  (d[:8], )
print(vid)

Output:

(['{"0": {"metadata": {"id": "fQ3JoXLXxc4"', ' "title": "| Board Questions | 12 Maths | Equivalence Class | Equivalence Class Board Questions |"', ' "tags": ["Board Questions"', ' "12 maths"', ' "12 maths Board Questions"', ' "Previous Year Board Questions"', ' "Maths Board Questions"', ' "Board questions based on Equivalence Classes"'],)

Now, my problem is: the cosine similarity matrix finds the cosine similarity between the set of documents in one tuple (ie. it finds the cos sim between all files in doc iir). How to compute the cos sim of each iir doc with each doc in the vid.

I tried to convert the vid json file to tuple, and then compute the cos sim between iir and vid, but it gives an error.

Can anyone help me with explaining how to find cos sim between files with different formats, how to loop through files and compute cos sim, how to rank docs on the basis of highest cos sim.


texts: g-mean for binary classification doesn't use sensitivity of each class?

texts:
scikit-learn's contrib package, imbalanced-learn, has a function, geometric_mean_score(), which has the following in its documentation:

  The geometric mean (G-mean) is the root of the product of class-wise
  sensitivity. This measure tries to maximize the accuracy on each of
  the classes while keeping these accuracies balanced. For binary
  classification G-mean is the squared root of the product of the
  sensitivity and specificity. For multi-class problems it is a higher
  root of the product of sensitivity for each class.

Why is sensitivity and specificity used for binary classification? In the sources below, geometric mean  is defined as the geo mean of precision and recall.

Cross Validated answer

  g-mean is defined as $g = \sqrt{\ Precision * Recall\ }$

Towards DS: Beyond Accuracy

  There are other metrics for combining precision and recall, such as the Geometric Mean of precision and recall, but the F1 score is the most commonly used.


texts: Experiencing unexpected output from Knime Naive Bayes Predictor

texts:
I'm working on a single-class series with knime and the naive bayes learner/predictor set.

however when I run a prediction on this, the output of the predictor is only ever 0 or 1 and very closely (100%) resembles the value of the other training column (IncidentOnDate) rather than the target variable (NumIncidents2).

I'm new to DM and Knime, so would be grateful if someone could point me in the right direction. I would expect to see output that resembles the same probability distribution and I am intending to get to a point where i can forecast the next 30 days based on distribution probability from the previous periods (I have 2500d of data).


texts: Chi-square test - how can I say if attributes are correlated?

texts:
I am experimenting a course's teorical contents on this dataset. After data cleaning, I am trying to use chi-square test. I wrote the following code:

chisq.test(chocolate$CompanyMaker, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$SpecificBeanOriginOrBarName, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$CompanyLocation, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$BeanType, chocolate$Rating, simulate.p.value = TRUE)
chisq.test(chocolate$BroadBeanOrigin, chocolate$Rating, simulate.p.value = TRUE)

chisq.test(chocolate$CompanyMaker, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$SpecificBeanOriginOrBarName, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$CompanyLocation, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$BeanType, chocolate$CocoaPerc, simulate.p.value = TRUE)
chisq.test(chocolate$BroadBeanOrigin, chocolate$CocoaPerc, simulate.p.value = TRUE)

And these are my results:

RATING

CompanyMarker   = 0.29
Specific...     = 0.6267
CompanyLocation = 0.1819
BeanType        = 0.5372
BroadBeanOrigin = 0.1534

COCOA PERC

CompanyMarker   = 0.0004998
Specific...     = 0.902
CompanyLocation = 0.04748
BeanType        = 0.8136
BroadBeanOrigin = 0.8356

Online, I read about significance level, but i didn't quite understand it. In particular, is it at 0.5 or 0.05? Which values are "ok"?

From what I understood, I should say that CompanyMarker, CompanyLocation and BroadBeanOrigin are related to Rating, while CompanyMarker and CompanyLocation are related to cocoaPercent.

Is this right?
If not, can you write or link me an example or a guide to do it right?
Thanks.


texts: Supervised Learning could be biased if we use obsolete data

texts:
What if the data that we could use for the training is obsolete. For instance, if I train my model with the computer sales report from the 20th century and try to predict the actual trends, a disaster, right?

Another good example is the one that profiles criminals. It will use the historical "mistakes" as fact and could wrongly suspect innocents(based on race or ethnicity).

How could I avoid this kind of situations?


texts: Should you use random state or random seed in machine learning models?

texts: I'm starting to study machine learning. All the examples I saw, the person that created the ML model used a random state or a random seed to stop the randomness of the process. But, in real life, when you're trying to apply a machine learning model into an actual project of a company, should you use any random state or seed? Is it right (in data science terms) to set a random state to the machine learning model and make reproducible results?


texts:
I have a time series data. It has daily frequency.

I want to forecast the data for the next week or month with an ARIMA model.

This is a chart of my time series data:

First I use the method seasonal_decompose from stats model to check the trend/sessionality/residual looks like:

from statsmodels.tsa.seasonal import seasonal_decompose
result = seasonal_decompose(df['n_transactions'], model='add')
result.plot();

I check if my data is stationary:

from statsmodels.tsa.stattools import adfuller

def adf_test(series,title=''):
    """
    Pass in a time series and an optional title, returns an ADF report
    """
    print(f'Augmented Dickey-Fuller Test: {title}')
    result = adfuller(series.dropna(),autolag='AIC') # .dropna() handles differenced data

    labels = ['ADF test statistic','p-value','# lags used','# observations']
    out = pd.Series(result[0:4],index=labels)

    for key,val in result[4].items():
        out[f'critical value ({key})']=val

    print(out.to_string())          # .to_string() removes the line "dtype: float64"

    if result[1] <= 0.05:
        print("Strong evidence against the null hypothesis")
        print("Reject the null hypothesis")
        print("Data has no unit root and is stationary")
    else:
        print("Weak evidence against the null hypothesis")
        print("Fail to reject the null hypothesis")
        print("Data has a unit root and is non-stationary")

adf_test(df['n_transactions'])

Augmented Dickey-Fuller Test:
ADF test statistic       -3.857922
p-value                   0.002367
# lags used              12.000000
# observations          737.000000
critical value (1%)      -3.439254
critical value (5%)      -2.865470
critical value (10%)     -2.568863
Strong evidence against the null hypothesis
Reject the null hypothesis
Data has no unit root and is stationary

I use auto_arima in order to get the best parameters for my model:

from pmdarima import auto_arima
auto_arima(df['n_transactions'],seasonal=True, m = 7).summary()

I train my model with this paremeters:

train = df.loc[:'2020-05-12']
test = df.loc['2020-05-13':]

model = SARIMAX(train['n_transactions'],order=(1, 1, 1))
results = model.fit()
results.summary()

I calculate the predictions:

start=len(train)
end=len(train)+len(test)-1
predictions = results.predict(start=start, end=end, dynamic=False, typ='levels').rename('SARIMA(0,1,3)(1,0,1,12) Predictions')

ax = test['n_transactions'].plot(legend=True,figsize=(12,6),title=title)
predictions.plot(legend=True)
ax.autoscale(axis='x',tight=True)
ax.set(xlabel=xlabel, ylabel=ylabel);

But the model can't obtain good results, why?

Edit

I have used instead of counts the revenue that I obtain for this counts as you suggested me that may be this would be the problem:

But the model is not obtaining good results:

What conclusion can I extract from here?


texts: My corr() function in Python keeps resulting in an "ValueError: The truth value of a Series is ambiguous..."

texts:
I am a very inexperienced programmer, this is my first question on the Data Science StackExchange, I sorry if it is formatted poorly or comes across as basic. For some strange reason, in Python, whenever I try to run a correlation function on the population density & total cases per million columns of my COVID-19 DataFrame (which I imported/read into Spyder as a csv), I keep getting the same long error message, namely, "ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()." regardless of whether I use the correlation function from Pandas or Numpy.
My first thought was that this error was caused by the presence of null values in those columns, so I used df.dropna(), then ran the correlation function again but I got the same "ValueError", so I have no idea what is going on here, I was able to run a correlation on those same columns just fine in RStudio which I am equally unskilled and inexperienced with.


texts: where does the downloaded file gets stored in the R-Studio Server on Ubuntu EC2 Instance when using download.file()

texts:
Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client.For the Purpose of this Analysis i first want to download a file from the Website called UCI Machine Learning Repository and then access the file for downstream Analysis, so i used the Below shown R Script in the r-studio server in my browser

download.file("https://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data", destfile = "Haberman.txt")

This works exactly fine but since i am new to work in the Cloud Platform i couldn't understand where RStudio stores the Haberman.txt file, because when i work on my PC i used to specify the Path to store the Haberman.txt file in the destfile argument as shown below

 download.file("https://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data", destfile = "E:/Couresera_Datasets/Haberman.txt")

Now when i Navigate to the specified Path in my PC i will be able to find the Haberman.txt file but where can i find the Haberman.txt file in the Ubuntu EC2 instance when i use the download.file()?

Any Help Will be Appreciated.

Thank you in Advance


texts: Proper way to make Train/test split on Time-Series

texts:
I want to create a model with LSTM to predict a user the next purchase value. For this I have used I used a user's purchase history. I have created the model and it works well, but honestly, I don't know I do the Train/Test split on the proper way or not.
To do this, I have used (univariate) user's purchase history.(X-purchase history values, y-target purchase value) As a first step, I have created a sliding-window process that creates new data. (As you can see in the pic) In the original dataset, I had 1000 users with 2820 timestamps and 1 feature (purchase values), with the Sliding-window process I got 1000*2320 users with 500 timestamps and 1 feature.
X.shape -> OriginalDataShape (1000, 2820, 1)
X.shape -> ModifiedDataShape (2320000, 500, 1)

# model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(500,1))) model.add(Dense(1)) model.compile(loss='mae', optimizer='adam')

# train
repeats = range(3)
scores = list()
for i in repeats:
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
    model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5, batch_size=32)
    pred = model.predict(X_test)
    score.append(metrics.mean_squared_error(pred,y_test))
print('Final score (Mse):')
print(score)

My questions are: It is the proper way or not? If it is not, do you have a suggestion or GitHub link for the solution?


texts:
Let's say one has many time series for which one wants to build a predictive model (based on LSTM). Which of the following cases would be more optimal and why?

1) building one model for all the time series in which each time series has a categorical label which will be included in the model by one hot encoding.

2) building individual models for each time series.


texts: Calculation of Neural network biases in backpropagation

texts:
While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training.

The initial code is without biases, which I'm trying to implement, forward and back calculations. From different internet sources I see that bias is just like other weights with a static input value 1, and backpropagation calculation should be similar and simplier.

But my current code version is not working - with the same input I get very different results from ~0.002 to ~0.99.

Please help me to fix biases calculations. Probably lines marked with ???. Here is a Python 2 testing code:

import numpy as np

# Sigmoid and it's derivative
def nonlin(x, deriv=False):
    if (deriv == True):
        return x*(1-x)

    return 1/(1+np.exp(-x))

X = np.array([[0,0,1],
              [0,1,1],
              [1,0,1],
              [1,1,1],
              [1,1,1]])

Y = np.array([[0],
              [1],
              [1],
              [0],
              [0]])

# Static initial hidd. layer weights for testing
wh = np.array([[-0.16258307,  0.43597283, -0.99471565, -0.39715906],
               [-0.70551921, -0.81601352, -0.62549935, -0.30959772],
               [-0.20477763,  0.07532473, -0.15920573,  0.3694664 ]])
# Static initial output layer weights for testing
wo = np.array([[-0.59572295],
               [ 0.74949506],
               [-0.95195878],
               [ 0.33625405]])

# Hidden layer's biases
biasH = 2 * np.random.random((1, 4)) - 1  # ???
# Output neuron's bias
biasO = 2 * np.random.random((1, 1)) - 1  # ???
# Static hidden layer's biases input
biasInputH = np.array([[1, 1, 1, 1]])     # ???
# Static output layer's bias input
biasInputO = np.array([[1]])              # ???

# Number of iterations to teach
for j in xrange(60000):

    # Feedforward
    h = nonlin(np.dot(X, wh) + biasH)
    o = nonlin(np.dot(h, wo) + biasO)

    # Calculate partial derivatives & errors
    o_error = Y - o

    if (j % 10000) == 0:
        print "Error:" + str(np.mean(np.abs(o_error)))

    o_delta =  o_error * nonlin(o,     deriv=True)
    o_biases = o_error * nonlin(biasO, deriv=True)  # ???

    h_error =  o_delta.dot(wo.T)
    h_delta =  h_error * nonlin(h,     deriv=True)
    h_biases = h_error * nonlin(biasH, deriv=True)  # ???

    # Update weights and biases
    wo += h.T.dot(o_delta)
    wh += X.T.dot(h_delta)

    # biasH += biasInputH.dot(h_delta)  # ???
    # biasO += biasInputO.dot(o_delta)  # ???

# Try new data
data = np.array([1,0,0])

print "weights 0:", wh
print "weights 1:", wo
print "biases 0:",  biasH
print "biases 1:",  biasO
print "input:   ",  data

h = nonlin(np.dot(data, wh))
print "hidden:  ", h
print "output:  ", nonlin(np.dot(h, wo))


texts: How to compare hyperparameter tuning in R and Python

texts:
I tried random forest in both R (Caret) and Python (Scikit-learn), but the results differ drastically.
Pearson correlation between predicted value and actual value was 0.2 in python whereas 0.8 in R.
I suspect that is because of hyperparameter tuning difference and wanted to check detail settings in R and python. Since I manually designated hyperparameters in Python explicitly, I know hyperparameter values in Python, but in R I am not sure which corresponding hyperparameters were used. In R, it only shows mtry was chosen as 46 and number of trees were 500.
In python, however, there were many more hyperparameters such as maximum number of levels in tree(max_depth), minimum number of samples required to split a node (min_samples_split), minimum number of samples required at each leaf node. Etc which I couldn’t find in R.
Are there hidden default settings for the values in R or R doesn’t consider hyperparameters other than mtry and number of trees? If so, why?
I tried to search such information but there was no information available.
Below are the links that I referred to.
https://arxiv.org/pdf/1804.03515.pdf
https://topepo.github.io/caret/train-models-by-tag.html#random-forest


texts: I have heard about the term Neural Network ZOOs, which are supposed to be repositories where there are a lot of pre-trained neural network models for many different applications, but I'm struggling to find them. Does anybody have some links for these kinds of repositories, especially for keras models?


texts:
The data table below contains cash flows going into an account. At index 0 the account starts with 50 in debt, is charged a monthly Rent of 90, and pays a certain amount every month, as well as getting additional supporting payments every 4 periods.

Assuming we know the first 8 periods (in order to form an opinion on the account, and to notice the seasonality of "Support"), how do we define the true balance based on previous and potentially future figures? The goal is to state how much someone actually has to pay, based on their current (and future forecasted) payment patterns.

For example, as below, if a "Support" transaction is missed in a week where it was expected, the true balance spikes up, but then if that payment was received, the true balance goes back down.

So far I've managed to confirm the following:

Rent is a known array (for past and future)
Payment is historically known and can be forecasted using ARIMA
Support is historically known and can be forecasted using Exponential Smoothing

I've tried to find a way to create a regression equation based on historical and forecasted values (so that the true balance takes into accounts expected payments), but so far I've not managed to do so. It feels like I'm hitting a brick wall and I would appreciate information if the way I'm attempting to do so is correct or not.

Note: The information provided here contains very simplified data. In reality, "Payment" can be highly irregular, while "Support" can either be a seasonal transaction that can be delayed by a number of periods, or can periodically stop and start again. I have so far been prototyping in Excel, as I'm unsure what to use to find a solution in Python.


texts: Is there any difference between feature extraction and feature learning?

texts:
It appears to me that "feature extraction" and "feature learning" are equivalent  concepts, however there are 2 separate wikipedia articles dedicated to them that are notably different. In particular, only in the Feature Learning article Neural Networks/Deep Learning are mentioned. However, it seems like they would be equally appropriate for either because autoencoders extract features from the raw (typically image) data and those now extracted features feed into the next layers.

So, what's the real difference between these terms?


texts: What is whole word masking in the recent BERT model?

texts:
I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it:

  In the original pre-processing code, we randomly select WordPiece tokens to mask. For example:

Input Text: the man jumped up , put his basket on phil ##am ##mon ' s head

Original Masked Input: [MASK] man [MASK] up , put his [MASK] on phil [MASK] ##mon ' s head

  The new technique is called Whole Word Masking. In this case, we always mask all of the the tokens corresponding to a word at once. The overall masking rate remains the same.

Whole Word Masked Input: the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head

I can't understand "we always mask all of the the tokens corresponding to a word at once". "jumped", "phil", "##am", and "##mon" are masked and I am not sure how these tokens are related.


texts:
So I'm doing the Udemy course for Data Science in Python, and there's something weird happening with the plots. The first one in the above picture is mine, the second is the instructors.

How do I get rid of the white space on the sides, they seem to be default when using the plotting function in pandas.

Edit:
This is all the code. Everything is basically default, so I'm guessing the matplotlib has some default settings changed in newer version of the package. The file df3 is here https://github.com/sxsheng/Udemy

Code:

import pandas as pd

import matplotlib.pyplot as plt

df3 = pd.read_csv('df3')

%matplotlib inline (This line is to be used in jupyter notebooks to display plots)

df3['a'].plot.hist()


texts: Timeseries prediction error measurement. How to deal with diffrent time scales?

texts:
I have some time series and a prediction model. Now I would like to measure how good/bad the prediction is for different products.
The problem is that for each product the time points (frequency of information varies very strong)

So as an example on series have updates in second intervals:

                               product price
          2014-08-25 01:01:01  A       1.2
          2014-08-25 01:01:01  A       1.3
          2014-08-25 01:01:02  A       1.2
          2014-08-25 01:01:03  A       0.9
          2014-08-25 01:01:04  A       1.4

The other series might have updates in days:

          2014-08-25 05:00:00  B       10.2
          2014-08-26 06:00:00  B       10.2
          2014-08-27 06:00:00  B       7.2
          2014-08-28 21:00:00  B       10.2
          2014-08-29 22:00:00  B       10.2

On the other hand, the prediction continues. So at each time point, the model outputs a price. Right now I tried to resample everything to 60 seconds interval using last prices and take the SMAPE  measure since the prices are always greater than 1. I don't like the results yet since the not that frequent time series tend to get bad results compared to more frequent time series. Maybe there is a good way to deal with that. Especially if the error measure could take into account that if the time series is more irregular, the measurement should be more relaxed.

I'm thankful for any Idea or hint.


texts: How to match features in new records for NLP BOW

texts:
I have a dataset that has 100,000 records
data in this dataset are 2 columns
1- Text
2- Class
When I apply BOW of my model I get big list of features
That is fine, I managed to work with them
my problem is after building the model and deploying.
now if a new text came with new words then the model wont work as it wokds in same feature structure
Example
"This is a test, test is important" , Red
"Adam pass a test", Green
so my final dataset is
This is a test important Adam pass class
 1    2 1 2    1          0    0   Red
 0    0 1 1    0          1    1   Green

once model created and got this text
"test and exam are similar", Yellow
in this case the set of features has new features which are
and exam are similar
the model will break coz these features never included in the training model
I wonder how to resolve this issue?


texts: How is dimensionality reduction achieved in Deep Belief Networks with Restricted Boltzmann Machines?

texts: In neural networks and old classification methods, we usually construct an objective function to achieve dimensionality reduction. But Deep Belief Networks (DBN) with Restricted Boltzmann Machines (RBM) learn the data structure through unsupervised learning. How does it achieve dimensionality reduction without knowing the ground truth and constructing an objective function?


texts: How to learn a classifier from a dataset with high imbalance

texts:
What are the most useful techniques for learning a binary classifier from a dataset with a high degree of imbalance (i.e., a dataset with the "target" class being much rarer than the "background" class)? For example,

Should one first down-sample the majority/background class to reduce its frequency and then readjust the probabilities reported by the learning algorithm? How should one do the readjustment?
Should one use different approaches for different learning algorithms, i.e., are there different techniques for dealing with imbalance in SVM, random forests, logistic regression, etc.?


texts: Are there separate parameters/weights in LSTM cells for each timestep?

texts:
As I understand, the parameters and weights of a basic RNN is the same for each time step - there is only 1 set to train. Is this also the case for standard LSTM cell?

More specifically, these are the equations for the standard LSTM cell on Wikipedia:

\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}

Are there separate/distinct $W$'s and $U$'s for each timestep?


texts: Number of Fully connected layers in standard CNNs

texts:
I have a question targeting some basics of CNN. I came across various CNN networks like AlexNet, GoogLeNet and LeNet.
I read at a lot of places that AlexNet has 3 Fully Connected layers with 4096, 4096, 1000 layers each. The layer containing 1000 nodes is the classification layer and each neuron represents the each class.
Now I came across GoogLeNet. I read about its architecture here.
It says that GoogLeNet has 0 FC layers. However, you do need the 1000 node layer in the end with Softmax activation for the classification task. So the final layer isn't treated as FC in this?
Also then, what is the number of FC layers in LeNet-5?
A bit confused. Any help or leads would be greatly appreciated.


texts: Pattern Recognition, Bishop - MLE and Least squares section 3.1

texts:
Please refer to an equation in Pattern recognition and Machine Learning by Bishop. My query is related to manipulation of an error term, equation 3.18:
$$E_{D}(\boldsymbol{w}) = \frac{1}{2}\sum_{n=1}^N\{t_{n} - w_{0} - \sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})\}^{2} \tag{3.18}$$
the author goes on to get:
$$w_{0} = \overline{t} - \sum_{n=1}^{M-1}w_{j}\overline\phi_{j} \tag{3.19} $$
$$where \quad \overline{t} = \frac{1}{N}\sum_{n=1}^Nt_{n} \qquad \overline{\phi_j} = \frac{1}{N}\sum_{n=1}^{N}\phi_{j}(x_{n}) \tag{3.20}$$
If I take the derivate of 3.18, w.r.t $w_{0}$, set it to zero, and divide both sides by $N$, I get:
$$w_{0} = \frac{1}{N}\sum_{n=1}^{N}t_{n} - \frac{1}{N}\sum_{n=1}^{N}\sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})$$

Am I creating any error above? If so, please help. If not, my problem is related to the second term (R.H.S., above equation):

  The right term,
  $\frac{1}{N}\sum_{n=1}^{N}\sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})$, is the
  average of double sum. I can't understand how we can simply 'push'
  $\frac{1}{N}$ inside to get
  $\sum_{j=1}^{M-1}\frac{1}{N}\sum_{n=1}^{N}w_{j}\phi_{j}(x_{n})$ and
  eventually equation for $\overline\phi_{j}$ (ref. 3.20), since:

$$\frac{1}{2}[(1+2) + (3+5)] \quad != 1 + 3 + \frac{1}{2}(2  + 5)$$

LHS is average of sum of sums and RHS is average 'pushed' inside.


texts: Representing the architecture of a deep CNN

texts:
Suppose I am feeding a $60\times60$ RGB image as the input to deep CNN with the first layer created using the following Keras code model.add(Conv2D(64, (3, 3), input_shape=(60, 60, 3))).

Will 64 filters be created for each channel (red, green and blue) of the image?
Is the following representation of the network correct? If 64 filters are created for each channel, then should I write "3X64@58X58"?


texts: Softmax function result for already normalized probabilities

texts:
Isn't the aim of softmax function normalizing the probabilities such that they all sum to 1? So when we apply this method to the already normalized numbers, it would change them. what do these new outcomes depict? Since they are changed, are they reliable?

from scipy import special
scipy.special.softmax([0.4,0.6])

gives:

 array([0.450166, 0.549834])


texts:
I am writing my masters thesis and am using LSTMs for daily stock return prediction. So far I am only predicting numerical values but will soon explore a classification style problem and predict whether it will go up or down each day.
I have explored several scenarios

A single LSTM using as input only the past 50 days return data
A stacked (2 layers) using as input only the past 50 days return data

The results are not great for either (and I didn't expect them to be). So I tried some feature engineering using 3 day MA, 5 day MA, 10 day MA, 25 day MA, 50 day MA of the daily returns as well as the actual daily return, meaning I have 6 input features. All other variables are kept constant yet the model now overfits (see the training and test loss plots below). Does anyone have any ideas why this may be?

Test Loss in orange and Train in blue


texts: Size Issue when Plotting the Predicted Vaue

texts:
I wrote a simple Stock Prediction Algorithm and got the predicted value. Then, I wanted to plot the relation between Adjusted close price and predicted value, but got the ValueError: x and y must be the same size. I tried to reshape it, but no luck. I'm having problem with the last 5 lines of the following code.
How can I resize X and y_predict in order to have a same size of them? What is the exact problem with my resize code?

import quandl
import pandas as pd
import numpy as np
import datetime
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.linear_model import LinearRegression
from sklearn import preprocessing, svm, datasets
from sklearn.model_selection import cross_validate, train_test_split

stock_data = quandl.get("WIKI/AAPL")
stock_data = stock_data[["Adj. Close"]]
forecast_out = int(30)
stock_data["Prediction"] = stock_data[["Adj. Close"]].shift(-forecast_out)
X = np.array(stock_data.drop(['Prediction'],1))
X = preprocessing.scale(X)
X_forecast = X[-forecast_out]
X = X[:-forecast_out]
y = np.array(stock_data["Prediction"])
y = y[:-forecast_out]
LinearRegression().fit(X.reshape(-1,1),y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = 0)
lin_reg = LinearRegression().fit(X_train,y_train)
y_predict = lin_reg.predict(X_test)

LinearRegression().fit(X.reshape(-1,1),y_predict)

plt.scatter(X_train, y_predict,  color='blue')
plt.plot(X_train, regr.coef_[0][0]*X_train + lin_reg.intercept_[0], '-r')
plt.xlabel("Adjusted closing price")
plt.ylabel("Predicted price")
```


texts:
In the research paper, for S=7,B=2, the model predicts 2 bounding boxes for every 7x7 grid cell hence 7x7x2=98 images are predicted per image. Yet the demo output image only has 3 boxes. Why is that?

My theory is that since thickness of the lines is proportionate to the confidence scores of the bounding box, after the model is trained, the "lousy" bounding boxes are so thin that they don't even appear.

The paper also says "Often it is clear which grid cell an object falls in to and the network only predicts one box for each object".

I'm confused.


texts: value error of incompatible wrong dimension during training a model

texts:
I was implementing a decision tree on a dataset. Before that, I wanted to transform a particular column with CountVectorizer. For this, I am using pipeline to make it simpler.
But there is an error of incompatible row dimensions.
code
# Imported the libraries....
from sklearn.feature_extraction.text import CountVectorizer as cv
from sklearn.preprocessing import OneHotEncoder as ohe
from sklearn.compose import ColumnTransformer as ct
from sklearn.pipeline import make_pipeline as mp
from sklearn.tree import DecisionTreeClassifier as dtc

transformer=ct(transformers=[('review_counts',cv(),['verified_reviews']),
                             ('variation_dummies', ohe(),['variation'])
                            ],remainder='passthrough')

pipe= mp(transformer,dtc(random_state=42))

x= data[['rating','variation','verified_reviews']].copy()
y= data.feedback

x_train,x_test,y_train,y_test= tts(x,y,test_size=0.3,random_state=42,stratify=y)
print(x_train.shape,y_train.shape)             # ((2205, 3), (2205,))

pipe.fit(x_train,y_train)                       # Error on this line

Error
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-79-a981c354b190> in <module>()
----> 1 pipe.fit(x_train,y_train)

7 frames
/usr/local/lib/python3.6/dist-packages/scipy/sparse/construct.py in bmat(blocks, format, dtype)
    584                                                     exp=brow_lengths[i],
    585                                                     got=A.shape[0]))
--> 586                     raise ValueError(msg)
    587
    588                 if bcol_lengths[j] == 0:

ValueError: blocks[0,:] has incompatible row dimensions. Got blocks[0,1].shape[0] == 2205, expected 1.

Questions

How is this error of incompatible row dimension forming?
How it can be solved?


texts: How to train-test split and cross validate in Surprise?

texts:
I wrote the following code below which works:

from surprise.model_selection import cross_validate

cross_validate(algo,dataset,measures=['RMSE', 'MAE'],cv=5, verbose=False, n_jobs=-1)

However when I do this: (notice the trainset is passed here in cross_validate instead of whole dataset)

from surprise.model_selection import train_test_split
trainset, testset = train_test_split(dataset, test_size=test_size)
cross_validate(algo, trainset, measures=['RMSE', 'MAE'],cv=5, verbose=False, n_jobs=-1)

It gives the following error:

AttributeError: 'Trainset' object has no attribute 'raw_ratings'

I looked it up and
Surprise documentation says that Trainset objects are not the same as dataset objects, which makes sense.

However, the documentation does not say how to convert the trainset to dataset.

My question is:
1. Is it possible to convert Surprise Trainset to surprise Dataset?
2. If not, what is the correct way to train-test split the whole dataset and cross-validate?


texts:
I would like to extract the 20 most informative features of a very large set of features $X$ coming from a dataset containing clinical data  by using the RFE function from scikit-learn in Python.

$X$ is a 68 x 1140 matrix where

Each row represents a recorded session.
For each subject, there are 4 recorded sessions.
Then, there are 17 subjects in the dataset.

My idea is to use 70% of the dataset (i.e. 70% x 1140 random features from each recording) and extract 50 features out of the whole dataset.

$Y$ represents a ranking from 0 to 2.

In other words, my data looks like this:

And my implementation in the code is the following:

## X = features
## Y = labels
p = 0.7
n_perc = round(X.shape[1]*X.shape[0]*p) #70% of the data -> number of elements (height x width x 70%)
rand_idx = np.random.randint(X.shape[1]*X.shape[0], size=n_perc) #random indices (70% of the data)
X_rnd = X.flatten()[rand_idx] #select that 70% in X
Y_rnd = np.repeat(Y,round(X.shape[1]*p)) #we match the dimensions for X_rnd - Y_rnd
selector = RFE(estimator=LogisticRegression(C=1),n_features_to_select=20) #run RFE
selector.fit(X_rnd.reshape,Y_rnd) #select best features

The idea is that I flatten all the values from X and I get only 70% random elements from $X$, i.e $X_{rnd}$ (and also adapt $Y$ accordingly, i.e. $Y_{rnd}$).

ValueError: Expected 2D array, got 1D array instead:
array=[-0.25367578  0.8069118  -0.63161352 ...  0.5500815  -0.37418711
0.2580666 ]. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

But some reason I'm getting this error, which I don't understand. It says that I should reshape the array if I have either one feature or one sample, but it's not my case.

Does anybody know what I should do? Is this how I should approach the problem? Should I reshape $X$ in another manner?

Thanks.


texts:
Reading an article about 1x1 convolution, I found this:

  It should be noted that a two step convolution operation can always be combined into one, but in this case [GoogLeNet] and in most other deep learning networks, convolutions are followed by non-linear activation and hence convolutions are no longer linear operators and cannot be combined.

What do they mean saying "two step convolution operation can always be combined into one"?


texts: Creating ANN's (LSTM) for multiple datasets

texts:
I have three datasets, each representing time-series water quality data from three different regions (upper, middle, lower regions) of the same geographic area. I want to create different types of models, including statistical models (eg. ARIMA) and ANN (LSTM) to compare their performances.
In order to compare the overall performance of the LSTM to the other statistical models, would it still be valid to train and create a different LSTM for each regional dataset or does the model have to be the exact same for each regional dataset in order to be able to directly compare the LSTM's to each ARIMA? Would it be better to just combine all the data, or should I choose a single regional dataset to experiment on?


texts: Handling large word embedding matrix in Python

texts:
I have a pre trained Glove word embedding matrix (U) of dimension (400000 * 50). Now for the purpose of query expansion I need to perform the operation matmul(U*U.T). This is the term by term similarity matrix. But this leads to a matrix size of (400000*400000) and Python is throwing me memoryError.

Is there a work around to this problem other than using a better machine ?


texts: AdaBoost.R2 learning rate from scikit learn

texts:
AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar.
The implementation of regression for AdaBoost in scikit learn uses this algorithm (paper is cited in the sources of the AdaboostRegressor class).
The thing is that there is a step fundamentaly different from the original version of Drucker. There is the introduction of a new parameter named 'learning rate' for the AdaBoost algorithm. I will use $\eta$ as notation for this parameter.
Looking at the code, I was able to see that the step calculating the model importance :
$\beta=\frac{\hat{L}}{1-\hat{L}}$
becomes :
$\alpha=\eta ln(\frac{1}{\beta})$
where $\alpha$ is the importance used in the implementation.
The modification of the weights also have been changed from :
$w_{i+1}=w_{i}\beta^{[1-L_{i}]}$
to :
$w_{i+1}=w_{i}\beta^{\eta[1-L_{i}]}$
This parameter is used to control the amound the weights are changed at each iteration, and also the model importance. The effect of this is that if you set a learning rate inferior to 1, then more models will be built before reaching an average loss of 0.5 (hence stopping the algorithm).
However I don't understand how this works mathematically and if someone kind could explain it a bit more it would be really great.
I also would love to know especially if setting a learning rate of 1 is equivalent to the original version of the algorithm.


texts:
I am searching for an automated method (ideally a python package) that produces a score to assess the credibility of a given text (e.g. from a webpage).

I am not searching for:

text complexity assessments (i.e. how long sentences are and how many difficult words are used) as for example flesch reading ease, smog
index, flesch kincaid grade, coleman liau index, automated
readability index, dale chall readability score, difficult words
index, linsear write formula, or gunning fog.
text coherence (i.e. how well the next sentence fit with the previous one) as for example Text Coherence Analysis Based on Deep Neural Network

Why is complexity/coherence not the same credibility? Because many texts advertising for example homeopathy use long complex scientifically sounding and complex word loaded sentences while being nonsense in terms of trueness. Therefore I am wondering if there is any method to assess the credibility/reliability of a given piece of text/webpage information automatically?


texts: How does tensor product/multiplication work in TensorFlow?

texts:
In Tensorflow, I saw the following example:

import tensorflow as tf
import numpy as np

mat_a = tf.constant(np.arange(1,13, dtype=np.int32), shape=[2,2,3])
mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])
mul_c = tf.matmul(mat_a, mat_b)

with tf.Session() as sess:
   runop = sess.run(mul_c)
   print(runop)

[[[ 88  94]
  [214 229]]
 [[484 508]
  [642 674]]]

How does the tensor multiplication work?


texts: Which One is the Best Way to Create Training Sequences for LSTM-based Class Prediction on Time-series Data?

texts:
Let's say I have time-series data in the following way. I need to create training sequences of a fixed length as an input to my LSTM model on PyTorch.
# Time, Bitrate, Class
  0.2,  312,     1
  0.3,  319      1
  0.5,  227      0
  0.6,  229      0
  0.7,  219      0
  0.8,  341      1
  1.0,  401      2

Once trained, the model should be able to predict classes of test data without classes. I am not sure how to create the training sequences. Let's say I define a sequence_length = 3 for the data snippet above. I have two approaches:

By shifting a window of sequence_length. In this case, the sequences:

# Sequence,                    Label of the Sequence
  [(312,1), (319,1), (227,0)]  0
  [(319,1), (227,0), (229,0)]  0
  [(227,0), (229,0), (219,0)]  0
  [(229,0), (219,0), (341,1)]  1
  [(219,0), (341,0), (401,2)]  2

The label of the first sequence is 0 (since the last element's class is 0). However, this seems not okay, because even if the last label is 0, the former two are not.

Fetching class by class up to sequence_length. if any of them is less than the sequence_length, I pad with the last element (since all tensors should have the same shape):

# Sequence,                   Label of the Sequence
 [(312,1), (319,1), (319,1)]  1   <- padded with the last element (319,1)
 [(227,0), (229,0), (219,0)]  0
 [(341,1), (341,1), (341,1)]  1   <- padded with the last element (341,1)
 [(401,2), (401,2), (401,2)]  2   <- padded with the last element (401,2)

Instead of right-padding with the last element, would zero padding from the left make more sense? If so, do I need to do something to make LSTM ignore those padded zeros?
Thanks!


texts:
What are some good starting points for learning medical image analysis and combining it with deep learning?

I would like to analyze images with bone cancers but not sure what is proper way to preprocess them and prepare for the model.


texts:
I have two lists. One is target and another is predicted values in a binary classification. The target is always zero. And I have to plot the results of the prediction against target in such a way, that when I get a value of 1 in prediction, then i need to plot it with a black dot on a square plane. The plane should contain white area (which means it has both target and prediction as zero) and black dots (in case of target zero but prediction as one). How can I achieve this? Thanks for the help in advance.
Something like this :


texts: How to figure out if the problem is time series Forecasting or not?

texts:
Recently I have encountered a time-series based, where I have a dataset which contains data for call-centers performance. The dataset contains information about the number of calls made by customers for each date which is further categorized region wise.

Time        Region  Number of Calls
2018-03-01  X       1245
2018-03-02  Y       1390

The task at hand is to forecast the overall performance of call-centers as well as forecasting each region performance as well for the next 3 months to see which regions will perform better than others.
I have a dataset with dates of June-2017 to Current Date

Questions

Is the data enough to perform a time-series analysis with credible
results for the next quarter?
How should I tackle the task to forecast Region Wise Performance as well


texts:
I have found some research papers specifying explicitly the normalisation  technique they used to get the results.

What difference do

IMG /255.0

And

IMG - mean / STD

Has on the performance of the CNNs?


texts:
I am quite new to the data science field, and I was wondering if it is worth the effort to deep dive into Knime which looks quite powerful for many data science uses.

I know python and scikit-learn but it seems very efficient that I can do most of my data science projects (business related) with a simple GUI interface and achieve similar if not better results than coding.

The specific ways I will use Knime or the other:
- Advising retail firms about association rules
- Customer segmentation
- Analyzing retail data to find patters in customer behavior

Thank you, Dan


texts: scikit-learn: High / low value for C in SVM

texts:
I'm playing with scikit-learn. Looking into the user guide and documentation they say:

  A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly.

The default value is 1.0, which is used by most of the examples. I also found the value C=100.

What is meant by "low" and high for C?


texts: How can I discover topics in a social media data-set?

texts: I'm working on a project and i need to discover topics existing in a social media data set. For instance, i wanna extract the topics existing on 200K tweets. Any one recommend to me any machine learning algorithms?


texts: Dimensions of Transformer - dmodel and depth

texts:
Trying to understand the dimensions of the Multihead Attention component in Transformer referring the following tutorial https://www.tensorflow.org/tutorials/text/transformer#setup
There are 2 unknown dimensions - depth and d_model which I dont understand.
For example, if I fix the dimensions of the Q,K,V as 64 and the number_of_attention_heads as 8, and input_embedding as 512 , can anyone please explain what is depth and d_model?


texts: Linear Regression Coefficient Calculation

texts:
class LR:

    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.xmean = np.mean(x)
        self.ymean = np.mean(y)
        self.x_xmean = self.x - self.xmean
        self.y_ymean = self.y - self.ymean
        self.covariance = sum(self.x_xmean * self.y_ymean)
        self.variance = sum(self.x_xmean * self.x_xmean)

    def getYhat(self, input_x):
        input_x = np.array(input_x)
        return self.intercept + self.slope * input_x

    def getCoefficients(self):
        self.slope = self.covariance/self.variance
        self.intercept = self.ymean - (self.xmean * self.slope)
        return self.intercept, self.slope

I am using the above class to calculate intercept and slope for a Simple Linear Regression.  However, I would like to tweak it to make it work for Multiple Linear Regression as well, but WITHOUT using matrix formula $(XX^T)^{-1}X^TY$.

Please suggest.


texts:
I'm trying to predict a continuous target in an industrial context.
The problem I'm facing is that the some of the predictors have changed over time, for example the pressure in the machine was increased. This influenced some of the other predictors, but hasn't influenced my target.
As an example (in R formula notation):
$ Y \sim U_1$ The target depends on some unobservable variable
$ X_j ~ \sim U_1 + X_i$   One of my observed variables depends on the unobservable varible and another observed variable. Therefore $X_j$ is helpful for predicting $Y$
Now $X_i$ has changed a couple of times. This clearly hasn't influenced my target. But I'm also not really able to learn the relation $Y \sim X_j$ because $X_j$ has changed with $X_i$.
I know some of these dependencies for fact from physics. But there's no way I can fix this by hand because there are about 1000 variables.
When reading about data drift, it's always about how to adjust your existing model to a sudden change, but for me the changes already happened in the past.
The time periods where nothing has changed are too small to just use the latest batch, but it seems like just using the whole dataset without any adjustments doesn't work either.
Can anyone give advice on how to address this?
(Right now I'm using XGBoost but I'm open to other models)


texts: Combining several Multi-Output-Models into a single Multi-Output-Model

texts:
I'm trying to create a k-Nearest-Neighbor based model of 76-dimensional input data $I$ and 44-dimensional output data $O$.
Through domain knowledge I know that only certain input dimensions are related to certain output dimensions.

Let the mapping be as follows:

I[0:35]  -> O[0:20]
I[35:37] -> O[20:23]
I[37:76] -> O[23:43]

Is there a possibility to combine three separate k-Nearest-Neighbors models (all with their own hyperparameter settings) to achieve the desired setup?

End result should be fast computation of:

I -> [combined model] -> O

sklearn's MultiOutputRegressor only takes single output models, which is not what I'm looking for.


texts: Use LSTM to predict the proportion of steps with nonzero feature values

texts:
I am trying to do a simple regression for sequences. Each input $X_i$ is a $n=2000$ by 1 matrix, formatted as $n_i$ 0-s followed by $(n-n_i)$ 1-s. The output $y_i$ should be $n_i/n$, i.e. the proportion of steps with nonzero feature values.

The code for generating the data

ns = [1087, 280, 1013, 300, 311, 260, 960, 281, 417, 311, 102, 251, 748, 428, 802, 512, 61, 80, 1700, 693, 147, 577, 280, 196, 377, 281, 88, 359, 250, 330, 679, 201, 284, 1129, 151, 285, 722, 1014, 153, 202, 256, 99, 661, 415, 134, 269, 196, 137, 436, 617]
Xs = np.array([[[0] for i in range(2000-x)]+[[1] for i in range(x)] for x in ns])
Ys = np.mean(Xs,axis=(1,2))
trainX, testX, trainY, testY = Xs[:45], Xs[45:], Ys[:45], Ys[45:]

I tried to use LSTM and an output layer to do the prediction, i.e.

model = Sequential()
model.add(Masking(mask_value=0.00, input_shape=(trainX.shape[1],trainX.shape[2])))
model.add((LSTM(10,activation='tanh',unit_forget_bias=True,input_shape=(trainX.shape[1],trainX.shape[2]))))
model.add(Dense(1,activation='relu'))
model.compile(optimizer=adam(),loss='mse',metrics=['mae'],)
model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1)

Yet I found the model fails to predict what I want, instead it predicts every sequence with almost the same $y$ value. Just wondering if there's any solution to solve this?


texts: Xgboost : A variable specific Feature importance

texts:
I have a data set something like this:
data = [['Alex',10,13,1,0],['Bob',11,14,12,0],['Clarke',13,15,13,1],['bob',12,15,1,1]]
df = pd.DataFrame(data,columns = ["dealer","x","y","z","loss"])

I am trying to predict binary column loss, I have done this xgboost model. I got Overall feature importance. Now I need top 5 most important features dealer wise.
How to do that?
I have tried to use lime package but it is only working for Random forest.
If I get Feature importance for each observation(row) then also I can compute the feature importance dealer wise.
kindly help
Desired Output :


texts: User-based nearest neighbour implementation in R?

texts:
I am just starting to learn to use R and am not sure how to find the best packages yet. I am looking for a package that will allow me to calculate user-based nearest neighbours as an input for a recommendation.

My original data is approximately as below:

User    Item    Rating
Anna    Apple   2
Bob Apple   4
Carlos  Apple   1
Ethan   Apple   1
Gene    Apple   2
Anna    Banana  4
Dana    Banana  3
Ethan   Banana  5
Frank   Banana  5
Bob Carrot  1
Ethan   Carrot  2
Gene    Carrot  4

I have used the package reshape2 and dcast method to generate a different datastructure where users are rows, items are columns and the rating is in the value section (similar to an Excel pivot).

User    Apple   Banana  Carrot
Anna    2   4   n/a
Bob 4   n/a 1
Carlos  1   n/a n/a
Dana    n/a 3   n/a
Ethan   1   5   2
Frank   n/a 5   n/a
Gene    2   n/a 4

Is there any R package that will allow me to compute the similarity between users based on the user vectors? Preferably I would like to use the Jaccard coefficient as my similarity metric. I tried to google for R packages with user-based nearest neighbour implementations but came up empty.

Any help of alternative suggestions will be much appreciated.

Thanks!


texts: regression to model exponential distribution

texts:
I have a work order system. I found out that the work order completion times are exponentially distributed. Every ticket has some features. I understand that regression is "a line that best fits the points". In my case, I want the line to be exponential curve. Is it possible ? Or is CDF for exponential distribution better option ?

My use case is I want to predict the probability of a work order completing by deadline. I was thinking of querying sets of data through "group by" & then compute and store the lambda parameters. When a new ticket comes in, I will pull the corresponding lambda & use CDF to determine %age. I am wondering if an ML algorithm can simplify this process.

I have .net stack. I use Accord.net. I'm open to other .net libs.


texts: Is numpy.corrcoef() enough to find correlation?

texts:
I am currently working through Kaggle's titanic competition and I'm trying to figure out the correlation between the Survived column and other columns.  I am using numpy.corrcoef() to matrix the correlation between the columns and here is what I have:

The correlation between pClass & Survived is: [[ 1.         -0.33848104]
 [-0.33848104  1.        ]]

The correlation between Sex & Survived is: [[ 1.         -0.54335138]
 [-0.54335138  1.        ]]

The correlation between Age & Survived is:[[ 1.         -0.07065723]
 [-0.07065723  1.        ]]

The correlation between Fare & Survived is: [[1.         0.25730652]
 [0.25730652 1.        ]]

The correlation between Parent-Children & Survived is: [[1.         0.08162941]
 [0.08162941 1.        ]]

The correlation between Sibling-Spouse & Survived is: [[ 1.        -0.0353225]
 [-0.0353225  1.       ]]

The correlation between Embarked & Survived is: [[ 1.         -0.16767531]
 [-0.16767531  1.        ]]

There should be higher correlation between Survived and [pClass, sex, Sibling-Spouse] and yet the values are really low.  I'm new to this so I understand that a simple method is not the best way to find correlations but at the moment, this doesn't add up.

This is my full code (without the printf() calls):

import pandas as pd
import numpy as np

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/test.csv")

survived = train['Survived']
pClass = train['Pclass']
sex = train['Sex'].replace(['female', 'male'], [0, 1])
age = train['Age'].fillna(round(float(np.mean(train['Age'].dropna()))))
fare = train['Fare']
parch = train['Parch']
sibSp = train['SibSp']
embarked = train['Embarked'].replace(['C', 'Q', 'S'], [1, 2, 3])


texts: How to run a model stored as a .Rda file?

texts:
I am using an already written R code which has a line of code as shown below

model_predictors <- buildModel(flag, data, outcomeName, folder)
model<-model_predictors$model
predictorsNames<-model_predictors$predictorsNames
auc <- model_predictors$auc
save(model, file=paste(folder,studyName,'_model_', flag$model[1], '_', outcomeName,".Rda",sep=''))

As you can see in the last line of code, the (training) model is saved in folder in a .Rda file format with a naming convention _model_. Now, I have to use this model to test/evaluate on unseen dataset.

So now my question, I see a .Rda file in my folder path and I can load the .Rda file in Rstudio but how do I make inference from this model? I am new to R and can anyone help me with this please?

Can someone help me understand how to run this model please?


texts: Understanding projection layer for BLSTM

texts:
In many research papers there are 'projection layers' related to BLSTM layers. For example, from here:

"we trained an 8-layer BLSTM encoder including 320 cells in each layer
and direction, and the linear projection layer with 320 units followed
by each BLSTM layer"

I can't understand what this means and how it works.
Any help on this topic would be appreciated. Thanks in advance!


texts: Does my prediction improve when I use more, but worse classifiers?

texts:
I have a logical problem when programming my tumor identification algorithm.
In my data sample, I have tested multiple antibodies on tumors - to identify whether those tumors are good or bad. This is essentially a binary classification. Each antibody (respectively "classifier") showed to have a different explanatory / prediction power when used on its own.
My question in regards to the guideline I now need to program is:
Do I get a higher overall prediction power if I use all antibodies on a new tumor? Do classifiers with a worse explanatory still improve my overall prediction?
Or would only using the antibody with the "best" individual prediction power lead to a better result?
The resulting algorithm would then, step by step, test each antibody against each new tumor - each test would be done independently.
And I somehow can't wrap my head around what's better and whether I should simply use only the "best" antibody (i.e. with the best predictive power) or use various to improve my prediction result.


texts: Neural network for variable length data classification

texts:
How can I create a network which can predict labels of variable lengths data:

Training data:

label1: abcdeaefafere
label1: afdfdofdjfdjdfdofdj
label1: dffdpodfdajfdjdddfddfd

label2: reorefdfpreperpe
label2: rexcxfrerperuetupterer
label2: erfdfdrpoeregjroeptreereter
...
...

test data:

fdldjffdjfjdfd
xcdjeioreweoforpeeedfdfd
...
...

Please note these are sequences belonging to different classes and this is a classification problem. I am not trying to predict future data for these sequences.

Thanks for your insight.


texts: Learn smoothly varying mean and variance of a variable over a 2d domain

texts:
For a problem which I am working on at the moment, I'm interested in learning how the mean and variance of some response variable y changes with two independent variables x1 and x2 - i.e. for each coordinate in (x1, x2)-space I wish to have an estimate for $\mu_y$ and $\sigma_y$ in order to be able to approximately standardise new observations as they arrive.
I have enough domain knowledge to expect both the mean and variance of y to vary smoothly across this space (plus I can likely place bounds on the gradient at any point), which I'd like to use to be able to deal with previously-unseen x1, x2 pairs as they arise.
What techniques would you suggest for such a problem?


texts: What is the difference between continual learning and active learning?

texts: As per my understanding, active learning is a kind of continual learning. Is there any difference between them?


texts: Why do we need convolutions over volume in convolutional neural networks for image recognition?

texts:
In convolutional neural networks, we make convolutions of three channels (red, green, blue) with a filter of dimensions $k\times k\times 3$, like in the picture:

Each filter consists of adjustable weights, and can learn to detect primitive features, like edges. The filter can be different for each channel: one for R, another for G, yet another for B.

My question is: Why do we need separate filters for each channel?

If there's an edge, it will appear on each channel at the same place. The only case when this is not true is if we have lines that are purely red, green or blue which is possible only in synthetic images but not in real images.

For real images these three filters will contain the same information which means that we have three times more weights, so we need more data to train the network.

Is this construction redundant or I am missing something?

Isn't it more practical to detect features on a gray-scale image where we just need a 2-dimensional convolution? I mean, somewhere in the pre-processing step, the image can be separated into a gray-scale part and some other construction containing only a color information. Then each part is processed by a different network. Will this be more efficient? Do such networks exist already?


texts:
I have a program in which I use sequence to sequence approach as a prediction model  with attention. The problem is, while training, the accuracy is always decreasing at each epoch, like shown in the figure attached.

If anyone can tell me the reason, I will be soo thankful.


texts: Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments?

texts:
I'm looking for a tool to track the results of several experiments/iterations in machine learning.

Training a model can take days/weeks, so it's essential to track its performance and be able to easily replicate results and roll back to previous versions.
I'm looking for a tool that makes the process easy and streamlined like git did for version control.

Useful features would be:

open source with a license that allows commercial application
self-hosted [all the information should reside on computers we own]
python-friendly (ideally, tensorflow-friendly as well)
can store both hyperparameter values as well as references to specific datasets (the latter would be useful to track the effect of hard negative mining, etc.)
cluster-friendly
include basic plotting (it's useful to visualize learning curves)
automatic testing on multiple datasets
user tracking [like "blame" in git]


texts: Exporting R model to OpenCV's Machine Learning Library

texts:
I'm wonder if it's possible to export a model trained in R, to OpenCV's Machine Learning (ML) library format? The latter appears to save/read models in XML/YAML, whereas the former might be exportable via PMML. Specifically, I'm working with Random Forests, which are classifiers available both in R and OpenCV's ML library.

Any advice on how I can get the two to share models would be greatly appreciated.


texts: On the properties of Hyperbolic Tangent Kernel

texts:
How do Hyperbolic Tangent Kernels work? That is what is the intuition behind them? Can you provide proofs and examples for illustration?

Hyperbolic Tangent Kernels are defined as:
$$ K(x, x^\prime) = tanh\bigg(\alpha (x\cdot x^\prime) + c\bigg)$$

For example, for Gaussian RBF kernel, the intuition is that the support vectors affect the decision surface based on the locality of influence. What is the analog for Hyperbolic Tangent (Sigmoid Kernels)?

Some references on the Hyperbolic Tangent Kernels are:

Hsuan-Tien Lin and Chih-Jen Lin
Sabri Boughorbel, Jean-Philippe Tarel, Nozha Boujemaa


texts: How to approach the PlantVillage dataset?

texts:
I'm working on the PlantVillage dataset and i want to predict the type of the disease from the image of a leaf. The dataset is labeled in pairs (Type of the plant,Healthy/name of the disease). I'm wondering which approach from the following is better ? Should I build 2 neural nets, one for recognizing the type of the plant, and the second for the name of the disease. Or, should i build one neural net that predicts at the same time the name of the plant and the disease.
In my opinion the first approach is better because i'll have more data for each task while in the second approach i'll have more classes and less data for each class. However, there might be a correlation between the plant and the disease that might improve the performance of the model.


texts:
How to know the right direction in a noisy environment?

In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and moves in that direction and somehow it works.

I imagine that if there are lots of neurons, that there is a large space of possibilities.

I am electrical engineer, so I am used to encounter noise. I am also new to the topic of neural networks, so pardon me if this is a beginner's question.

I fear that the space of possibilities looks very noisy if I take too close a look. Gradient descent knows only the close look: You get the gradient at THIS micropoint.

Add noise, your derived vector probably points anywhere it wants.

How can I achieve a less fine grain approach, to find a global minimum in a noisy plane, then go down to a better resolution in order to find the local minimum inside the global minimum?

Yes, one probable solution would be to train the network on some points in the near neighbourhood and to estimate the noise from there plus make an average... but that takes a lot of trainings, and those are expensive.

Am I thinking too complicated here?


texts: How to discard trash topics from topic models?

texts:
I am undertaking text analysis of some twitter data. In the end I want to have a data that is interpretable. And so in the end I would like to reduce the data to relevant unit of analysis. Topic models seem a nice fit since they reduce the noise significantly, but still at the end there are quite a lot of trash topic lists which do not mean anything.What would be the best way to remove these topics. Is there a working example of this?

I am using LDA. Probably I will use Gensim. Its going to be unsupervised and its a simple twitter data. So the whole corpora will be 1500 tweets of 160 characters.


texts: Different R Markdown in same document for each output type

texts: In R Markdown, I would like to be able to conditionally include certain chunks depending on what I am outputting to. For example if I am knitting a PDF I would like to include a small bit of LaTeX to do a page break, but if I am knitting HTML from the same document, I would like to skip that, or perhaps include something else. So how can a document "know" where it is going, and selectively include things?


texts: splitting of training examples into the mini batch: what to do with the rest tiny mini-batch?

texts:
Lets assume I have 103 training examples. I want a mini-batch to be of the size 16. That means that there will be 6 mini-batches of the size 16 and one mini-batch of the size 7.

In the tensor flow one needs to specify the shape of the input:

x = tf.placeholder(tf.int32, shape=[batch_size], name='x')

which of course led to the following error:

Cannot feed value of shape (7,) for Tensor 'x_10:0', which has shape '(16,)'

So, what do I do with the mini-batch of size 7? Should I find the corresponding mini-batch size, that will create equal mini-batches? If so, how can I follow then the advice to create mini-batches of the power of two? or I should disregard the last mini batch?


texts: anaconda change 'base' environment permanently to a different environment 'myenv' to startup at terminal openings

texts:
I would like to cahnge the 'base' install environemnt that inisiates when I start a terminal to an a diferent environment that i have created called 'myenv' anaconda evironment.
I want that 'myenv' stats every thime wen I star the terinal.
I have turned off the 'base' environemnt from initaiating.
conda config --set auto_activate_base false
I am on Ubuntu 20.04


texts: Is there any paper introduce an intuitive method for clustering evaluation?

texts: I would like to use the most intuitive method like minimizing the within-cluster distance and maximizing the distance between neighbouring clusters, but not sure does this method have a name or something? Is there any related paper regarding it? The only method I'm using now is Silhouette score. Many thanks.


texts: How to decide the number of primary components for PCA

texts:
Background

Trying to identify the number of primary components to use (k) for PCA for MNIST aiming at 95%.

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1)

# Split data into training and test
X, y = mnist["data"], mnist["target"]
X_train, y_train = X[:60000], y[:60000]

COVERAGE=0.95

If I follow Coursera Machine Learning - Principal Component Analysis Algorithm it is 67.

from sklearn.preprocessing import StandardScaler
X_centered = StandardScaler().fit_transform(X_train - X_train.mean(axis=0))
covariance_matrx = X_centered.T.dot(X_centered)
U, s, Vt= sp.linalg.svd(covariance_matrx)

calculated_coverages = ((s ** 2) / (len(s) -1)).cumsum()
calculated_coverages = calculated_coverages / calculated_coverages[-1]
k = np.argmax(np.array(calculated_coverages) >= COVERAGE)
print("k-th component to cover {0} is {1}".format(calculated_coverages[k], k))

  k-th component to cover 0.9507022719172283 is 66

However, if I use explained_variance_ratio_ from scikit learn, it is 154.

from sklearn.decomposition import PCA
pca = PCA()
pca.fit(X_train)

contributions = pca.explained_variance_ratio_
coverages = pca.explained_variance_ratio_.cumsum()
k = np.argmax(coverages >= COVERAGE)

print("k-th primary compoent for 95% coverage is {}".format(k + 1))

  k-th primary compoent for 95% coverage is 154

When I look at scikit-learn/sklearn/decomposition/_pca.py, it looks the logic is the same.

    U, S, V = linalg.svd(X, full_matrices=False)
    # flip eigenvectors' sign to enforce deterministic output
    U, V = svd_flip(U, V)

    components_ = V

    # Get variance explained by singular values
    explained_variance_ = (S ** 2) / (n_samples - 1)
    total_var = explained_variance_.sum()
    explained_variance_ratio_ = explained_variance_ / total_var
    singular_values_ = S.copy()  # Store the singular values.

Question

Please help understand why they are different.

Related

Deciding on the number of components in PCA
Python scikit learn pca.explained_variance_ratio_ cutoff


texts: Hot Encode vs Binary Encoding for Binary attribute when clustering

texts:
I am planning to use data for a clustering problem that contains a column with a binary value BUY/SELL.

Should I be converting this attribute and assign it binary values (BUY=1, SELL=0), and keep it on the same column, thus reducing the number dimensions

OR

Hot encode the attribute (adding two columns BUY and SELL and putting 1 on the appropriate column)?

How do these two methods of nominal to numeric conversion affect the final model for popular clustering algorithms (K means, Hierarchical, etc...)


texts: Clustering without information about identifier

texts:
I have a data-set with different products and binary value if it was sold in a store or not. I looks like:

product_id  store_1 store_2 store_3 store_4 store_5 store_6
0   A   1   0   0   1   0   1
1   B   1   1   0   0   1   0

Is there any way to cluster these products with any information about the products itself?

One thought I had was to generate distance between products and then cluster the product X product matrix.
Is this problem sort of similar to market basket analysis?

Any guidance will help.


texts: Is there a process flow to follow for text analytics?

texts:
I am trying to draw a process flow (like a template) to be followed while on text analysis projects. So far, I've come up with this.
Text Analytics Steps

Data Collection

Acquire data
Convert data into plain text

Remove Duplicate Entries
Text Parsing and Extracting Features

Tokenization
Parsing

Remove HTML characters
Decode complex symbols to UTF-8
Spell check
Apostrophe look-up
Remove punctuation marks
Remove expressions / emojis
Split attached words
Slangs look-up
Remove URLs

Lemmatization / Stemming (Normalization of Tokens)
Parts-of-Speech Tagging

Text Filtering

Remove start-words
Remove stop-words
Remove irrelevant words based on frequency

Text Transformation

Bag of Words Representation
TF-IDF

Text Mining / Analysis (whichever analysis needed)

Text Categorization
Text Classification (supervised)
Topic Modeling (unsupervised)
Text Clustering
Similarity Analysis
Sentiment Analysis

Is this flow in the right order of steps?
What are the steps/sub-steps that I am missing?
Does the process flow look like a template or go-to flow chart when undertaking any text analytics project?
Edit: Updated process flow


texts: How do I convert a summation equation to a vector equation (backpropagation)?

texts:
$$a_j^l=\sigma(\sum_{k} {w_j}_k^l {a}_k^{l-1}+b_j^l)$$
$$a^l=\sigma( w^l {a}^{l-1}+b^l)$$
In a resource I have been reading, the above equations describe the activation of a neurone. They have the same function (I agree) but the first is a summation representation, whereas the second is this equation but in a vector form. My questions are, What did they do to remove the subscripts, and what is the process for converting summation equations to vector equations?


texts: Procedure for selecting optimal number of features with Python's Scikit-Learn

texts:
I have a dataset with 130 features (1000 rows) . I want to select the best features for my classifier. I started with RFE but Its taking too long, i done this:

number_of_columns = 130

for i in range(1, number_of_columns):
    rfe = RFE(model, i)
    fit = rfe.fit(x_train, y_train)
    acc = fit.score(x_test, y_test

Because this took to long, I changed my approach, and I want to see what you think about it, is it good / correct approach.

First I did PCA, and I found out that each column participates with around 1-0.4%, except last 9 columns. Last 9 columns participate with less than 0.00001% so I removed them. Now I have 121 features.

pca = PCA()
fit = pca.fit(x)

Then I split my data into train and test (with 121 features).

Then I used SelectFromModel, and I tested it with 4 different classifiers. Each classifier in SelectFromModel reduced the number of columns. I chosed the number of column that was determined by classifier that gave me the best accuracy:

model = SelectFromModel(clf, prefit=True)
#train_score = clf.score(x_train, y_train)
test_score = clf.score(x_test, y_test)
column_res = model.transform(x_train).shape

End finally I used 'RFE'. I have used number of columns that i get with 'SelectFromModel'.

rfe = RFE(model, number_of_columns)
fit = rfe.fit(x_train, y_train)
acc = fit.score(x_test, y_test)

Is this a good approach, or I did something wrong?

Also, If I got the biggest accuracy in SelectFromModel with one classifier, do I need to use the same classifier in RFE?


texts:
The keras.wrappers.scikit_learn module can be used to build KerasClassifier model.
Can Keras be used to build clustering models?
If it can be, are there any examples for that?


texts: Multi-dimensional regression with ensemble models?

texts:
I'm curious to know whether boosting, random forests or other types of ensemble models can perform multi-dimensional regression.
To be precise: That means multiple outputs (multi-dimensional labels) that are not intended for classification. For example predicting a persons height and weight from some data where there are two outputs each corresponding to one of height/weight (the labels and output values can be normalized if necessary). You can easily do this with neural networks of course. I am not interested in the trivial case where you simply have multiple models, one per regression output.
If yes, are there any libraries available that support this (the ones I've looked at do not)?


texts: Converting nominal data to numeric - is using dictionaries the right approach?

texts:
Currently I have a data-set with roughly 7 types of nominal data; these are things like "workclass" or "marital status". It is my understanding it is best to convert nominal data like this into numeric data. So, what I've done so far, as a simple experiment, is make up a dictionary for each nominal data type, so in our example of "marital status", it would be "Divorced":0, "Married":1, "Widowed":3, "Separated":4.

However, as I understand, there are some problems with this approach, as it implies that Divorced->Married are more related than Divorced->Widowed. I tried to arrange this data into a logical pattern, but for a lot of it its impossible. So here are my questions:

Is there a good approach to do this without running into the above problem, specifically in Python? I'm using Pandas DF's at the moment, and I saw an example online of something called Binary Vectorizers, but I'm not sure if that won't introduce too many attributes/columns to my data set.
When I convert a nominal value to an interval one, do I replace the  original (For example Workclass) with the Workclass_Number data? Do I only enter the number data into my models?
Are there any guidelines on the numbers that should be used? For example I used simple intervals of 1 for all of my data, starting at 0.


texts:
I am trying to connect with cassandra in R using RJDBC. When I execute

casscon <- dbConnect(cassdrv, "jdbc:cassandra://ipaddrs:9160/demodb")

I am getting

Error in .jcall(drv@jdrv, "Ljava/sql/Connection;", "connect", as.character(url)[1],
: java.lang.StringIndexOutOfBoundsException: String index out of range: -1

I can't figure out the problem. I need a solution for this.