texts: Is there an easy to use .net library for neural networks?

texts:
I know 'easy to use' is going to be subjective, so let me qualify the question a little. Is there a library or working scrap of code that I can essentially copy into my project, change the number of neurons in each layer, the number of layers, and the source of the inputs and then click run. I've written at least 15 of these myself, not a single one has worked properly - I need to see something working in a language I understand so I can get to grips with why I'm apparently a moron!

(I can post some of my own code if anyone is interested, but it's lengthy and not too related to the question)


texts: How can a ML algorithm learn to classify fake news?

texts:
I am new in Machine learning techniques and in fake news detection by using these algorithms (SVM, nn, logistic regression,..).
I would like to understand how an algorithm can learn from a training set which include news and fake news, what it will be necessary to have (target), what type of information can be relevant for a good analysis and learning ...
I read many papers and most of them do not mention the type of news used, the criteria for building the classifiers, but they only show the results of the prediction when a new news is generated.
This could be perfectly fine, but it could be also easily to say: I got this amazing results!, without understanding how these results were got.
I would appreciate if you could show me an example of how an algorithm can learn from text (not necessarily news, but also tweets, or something else would be fine).
Following discussion with Erwan: one of his previous answer partially has answered my question. However I would like to understand the following. One needs to have a corpus, then label news/tweets in fake/not fake, then run the model. But how the algorithm works on texts and takes relevant words or features for detecting fake news?
So my question is:
If I have a corpus on Trump, would the algorithm be able to detect fake news on Vitamin C, without any words (verb, adj, noun,..) in common between the two dataset, except stopwords?


texts: Scikit model is not able to predict sequence correctly

texts:
I am trying to create a regression model using scikit-learn for predicting car price. The input data are, car model(trim), kilometers used, past resale price of similar car and age of used car. I am trying to predict the future resale price of the car.

I have done the preprocessing of data. I have tried using ARDRegression, RandomForestRegressor and finally MLPRegressor. But the prediction model doesn't seem to predict well, the prediction results seem to fall outside the range of training data.

Example: if the Actual value(actual selling price) is 748077.0 but predicted value is 1352960

Probably I have done some mistake but I am not able to figure out what it is. I have simplified the code to focus on prediction part and have shared the link.

Could someone please guide me.

This is the google colab notebook link

Update: Inverting the index of the dataframe iresults in total change of prediction.


texts: Does a single word vector, trained using Word2Vec or a similar approach, carries information or meaning?


texts: Considerations to take into account when clustering

texts:
My idea is to use clustering to perform stock segmentation based on risk, building different risk levels that might adapt better to different kind of users.
Hence I have computed different risk indicators as stock variance, VaR, loss probability and Sharpe Ratio. My final dataframe with SP&100 stocks looks like this:
I am wondering if I do need to have some considerations regarding data preprocessing since this variables are highly correlated, all of them are computed by a formulae that uses variance or standar deviation and mean returns.
I have been doing some research and found that I might need to perform data normalization and PCA analysis but I would like someone to explain me why in order to understand what I am doing and if I need to take any other considerations.


texts: Kaggle API does not download the entire dataset

texts:
I am trying to download the statefarm distracted driver detection dataset using the API key on Colab:
!kaggle competitions download -c state-farm-distracted-driver-detection
However, all this does is just download a handful of images. The dataset is 4.0GB large.


texts: Pros and cons of pandas or R for longitudinal data?

texts:
Note: I believe this question is not off-topic because it meets all of the criteria for subjective questions that are allowed. I would be happy to rephrase or clarify if others disagree

I'm about to begin a two-year project which will predominantly involve longitudinal panel data.
I've found numerous questions and answers (Pros and Cons of Python and R for Data Science), and blog posts (https://www.quora.com/Which-is-better-for-data-analysis-R-or-Python | http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html), about the relative merits of R and Python+pandas for data science but nothing that discusses longitudinal data.

My question is therefore:

Which environment (Python+pandas or R) would you use for longitudinal data analysis, and why?

For example, I would love to see answers that:

Tell me how you have used one or both environments to solve a particular problem with longitudinal data
Which environment or package(s) you found easier to work with, and why
If you used a notebook such as beaker to work with multiple environments simultaneously.
If so, which environment did you use for which step(s) in the data analysis pipeline, and why?
If so, did this confer advantages over just using one language: would you use such an approach again?

I am not asking which one is best (we all know that such questions are never constructive).

I am more familiar with one environment than the other, but I'm not averse to learning new skills (and both share similar syntax anyway), so I'm not going to say which one: I want answers based on your experience, not my abilities.


texts: Practical example and working of Laplace Smoothing or Linear Interpolation in Natural Language Processing (NLP)

texts:
Let us suppose we have a document where

total_words = 50  (for example -> is,the,now,is,am,here,now)
total_unique_words = 40 (for example -> is,the,am,here,now)

How can we apply the Linear Interpolation/ Laplace Smoothening in the case of a trigram

for example

('he') has a count of 12
('he', 'is') has a count of 8
('he','is','here') has a count of 4

('is','here') has a count of 6

('is') has a count of 18
('here') has a count of 5

I think it is related to the count of the values. I have the formula in terms of lambda * P(x_i | x_i-1) but do not know how to implement here. Can someone please give me an  PRACTICAL example of how to implement the interpolation

Please make any necessary assumptions in the case as I am learning it the first time


texts: DeepLearning: does it make sense to have more nodes in the initial layer than inputs--for tabular data

texts:
I was looking at a couple of keras tutorials and something struck me as odd. So in these two tutorials, both on housing data, the number of nodes in the first layer of the network were larger than the number of columns in the dataset itself. NOTE: I am talking about tabular data and not image data or autoencoders. I understand that autoencoders will widen during the decoding phase, and I know that segmentation models will widen during the deconvolution phase. Both of these examples seem to be different than what I am seeing in these tutorials.

The first tutorial used data on Kings County, CA housing prices to create a simple neural network. In this case, the Kings County data only has 19 variables but the network itself has an initial dense layer of size 100 nodes. Here is the code:

def basic_model_1(x_size, y_size):
    t_model = Sequential()
    t_model.add(Dense(100, activation="tanh", input_shape=(x_size,)))
    t_model.add(Dense(50, activation="relu"))
    t_model.add(Dense(y_size))
    print(t_model.summary())
    t_model.compile(loss='mean_squared_error',
        optimizer=Adam(),
        metrics=[metrics.mae])
    return(t_model)

In the second example I was looking at Aurelion Geron's recent book on Keras and Tensorflow. In chapter 10 he uses some different California housing data to estimate a keras neural network. Again in that case he has 8 variables in the dataset, but he starts with a Dense layer of 30.

As a caveat, I did not see any columns get converted over to one-hot encodings or some sparse formulation.

This situation where the network starts wider than the data seemed odd to me, but perhaps that is just me coming from the statistics world. Can anyone explain this? Thanks.


texts: How to train and evaluate machine learning models with growing/changing datasets over time

texts:
Assume that you have a classification machine learning model, and you start with an initial dataset that contains 3 classes. You split the initial dataset into training/testing spits, you train the initial model and evaluate it, and then overtime, you collect more data for your dataset. Now you have more data that you want to add to you initial training dataset. The question is: How do you you organize you dataset and model training regiment so you can effectively quantify possible improvement between the initial model and the new model?
One possible solutions: if you split the initial training dataset into training/testing data, once you have more data, you just split that data and then add the respective new training/testing splits into the original splits. But this approach feels a little too simplistic, and it could potentially lead to drift in the dataset itself.
This is similar to doing regression testing but for machine learning models, but I am not sure im using the correct technical jargon for it.


texts: I’m a novice to machine learning. I've been trying out different neural network implementations in R, including the neuralnet package and the deeplearning function of the h2o package. For neuralnet, the default setting is one hidden layer with one hidden neuron. With this setting, the model takes several minutes to fit to my data. In the h2o package, the default is two layers with 200 neurons each, and the model takes only a few seconds. How is this possible? Are these two entirely different types of neural networks, or is the h2o implementation just much more efficient?


texts: LSTM data preparation with multiple independent observation runs

texts:
I'm struggling to wrap my head around how to correctly deal with multiple independent time series observations for training an LSTM. If I simply concatenate my observation runs, then there will be an incorrect disjoint between the last value and first value of adjacent sequences.

By way of hypothetical example:

Say I'm doing some sort of semi-stochastic time series experiment. Let's say every day from 9am-10am I measure sound levels once per second at an intersection and record the number of cars driving by at that moment. And say I do this for 10 days.

So my data from each day is three columns: (time (sec), dB, cars)

And I have 10 of these datasets; one from each day I took measurements. Call them obs_day01, obs_day02, ..., obs_day10

So far so good.

The time interval is always the same, so I can remove that from my data, leaving me with 10 independent observation series of format (dB, cars).

A first thought would be to concatenate these into one long series and then use TimeseriesGenerator in Keras. But this won't work, because the boundary of every concatenation is uncorrelated. i.e. The observation at 09:59:59 in obs_day01 is uncorrelated to the observation at 09:00:00 in obs_day02. So if I concatenate my datasets, the LSTM will learn a pattern at those edges that isn't there.

So a second thought becomes to write a custom generator to pass to model.fit() that generates its batches of samples by drawing sub-sequences from all 10 sets of observations, but ensuring that it never generates a training sequence that "crosses" the 09:59:59 - 09:00:00 boundaries. Something like:

def customGenerator(args): # Pseudocode
    while True:
        # create batch_size subsequences
        # of seq_len, sampling from all 10
        # observation sets, but ensuring that
        # individual sequences come only from
        # a single day's observations

So it seems like maybe that would work, but

1) I'm realizing I still don't understand how to approach the problem of multiple sequence samples with training LSTMs, and

2) This doesn't seem like it should be as complicated as I'm making it. In standard (non-recurrent) deep neural networks, we always train with many samples of the same "type," so surely RNNs are no different, which means there's something here I'm not grokking...

Hope that makes sense. Guidance much appreciated...


texts: Group similarity by high dimension vector comparison

texts: I have a dataset of 256 rows with 61 columns/variables. Each row should be considered a vector of dimension 61. If I randomly split it, by rows, in 2 groups, how could I prove that the 2 groups are similar? The origin of the data is biomedical and nonlinear approaches should be preferable.


texts: Standard(s) for data representing measurement times with their interval of validity?

texts:
Is there a standard for representing the time interval of a measurement?

Any actual measurement is made at a point in time with, perhaps, an interval over which it is considered valid.  For example, a census datum on a State's population for the year 1990 would be associated with the interval 1990/01/01T00:00:00.0000 duration 1 year.


texts: Should a bias neuron be connected to previous neurons?

texts: I'm trying to add a bias neuron to my neural network that uses the backpropagation algorithm. I'm trying to figure out how I should go about this, should I treat the bias neuron as a regular neuron? which means it's connected to the neurons on the previous layer?


texts: Actor Network Target Value in A2C Reinforcement Learning

texts:
In DQN, we use;
$Target = r+\gamma v(s')$  equation to train (fit) our network. It is easy to understand since we use the $Target$ value as the dependent variable like we do in supervised learning. I.e. we can use codes in python to train the model like,
model.fit(state,target, verbose = 0)

where $r$ and $v(s')$ can be found by model prediction.
When it comes to A2C network, things becomes more complicated. Now we have got two networks. Actor and Ctitic. It is said, the Critic network is not different from the what it is done in DQN. Only difference is now we have got only one output neuron in the network. So, similarly, we calculate the $Target = r+\gamma v(s')$ after acting via sampled action from $\pi(a|s)$ distribution. And we train the model with model.fit(state,target, verbose = 0) in python as well.
However, in Actor case it is so confusing. Now, we have another neural network which takes states as input and gives probabilities as the output by using softmax activation function. It is cool. But the point I stuck is,  the dependent variable to adjust the Actor network. So in python,
model2.fit(state,?,verbose=0)

What is the ? value to "supervise" the network to adjust the weights? and WHY?
In several resources I found the Advantage value which is nothing but the $Target - V(s)$.
And also there is something called as the actor loss which is calculated by,

Why?
Thanks in advance!


texts: Evaluation code understanding in wide and deep model of tensorflow

texts:
I'm doing some research on the wide and deep model developed by Google. And I got 2 questions on the training and evaluation code fragment. (Complete code see here):

  train_hooks = hooks_helper.get_train_hooks(
      flags_obj.hooks, model_dir=flags_obj.model_dir,
      batch_size=flags_obj.batch_size, tensors_to_log=tensors_to_log)

  # Train and evaluate the model every `flags.epochs_between_evals` epochs.
  for n in range(flags_obj.train_epochs // flags_obj.epochs_between_evals):
    model.train(input_fn=train_input_fn, hooks=train_hooks)

    results = model.evaluate(input_fn=eval_input_fn)

    # Display evaluation metrics
    tf.logging.info('Results at epoch %d / %d',
                    (n + 1) * flags_obj.epochs_between_evals,
                    flags_obj.train_epochs)
    tf.logging.info('-' * 60)

    for key in sorted(results):
      tf.logging.info('%s: %s' % (key, results[key]))

    benchmark_logger.log_evaluation_result(results)

    if early_stop and model_helpers.past_stop_threshold(
        flags_obj.stop_threshold, results['accuracy']):
      break

So my questions:

default setting for epochs_between_evals is 2, but how is it possible for evaluating 1 epoch after training for every 2 epochs? Apparently there is one model.evaluate() after every model.train(), isn't it?
Perhaps the train_hooks controls that in model.train() to break every 2 epochs, but the parameters of train_hooks don't include num_epochs or epochs_between_evals. I'm a bit confused.
What does model.evaluate() actually do? As I check the source code, it seems it's just a forwarding pass to get evaluation outputs. Any other work done behind the scene?


texts: Python library for segmented regression (a.k.a. piecewise regression)

texts:
I am looking for a Python library that can perform segmented regression (a.k.a. piecewise regression).

Example:


texts: Why Liblinear performs drastically better than libsvm linear kernel?

texts:
l have a dataset of dim=(200,2000) 200 examples and 2000 features. l have 10 classes.

l used sklearn  for both cases :

svm.svc(kernel=linear)
LinearSVC()

However LinearSVC() performs drastically better than svm with linear kernel. 60% against 23%.
l'm supposed to get the same or comparable results since they are fed with same parameters and data.

What's wrong ?

Thank you


texts: The effect of imbalanced distribution of data

texts:
I read on Google's ML website if I have classification dataset with a ratio of 90% for one classification and 10% of the data for another classification.

In that case, should I use the exact same percentage of data for each classification?
i.e. deleting around 80% of the dataset to make it 10% for each classification.

The reason is that Google said that the ML model will learn and then it is more likely to have a classification of the 90% and that won't provide good predictions. (i.e) The predictions might be biased towards a single label/feature.

My dataset is 90% to 10% but that is indeed the actual ratio and it is more likely to have the classification of the 90%

Shall I delete 80% of my data or keep it as is and let the ML learn that it is indeed more likely to have a classification of the 90%?


texts: Modeling pixel intensity with the normal distribution

texts: I was reading a machine learning book with applications in computer vision. In it, it mentioned that "in vision, it is common to ignore the fact that the intensity of a pixel is quantized and model it with continuous normal distribution." So my question is why do we choose to do that?


texts: Text Dataset Help: Need to figure out how to get the publication date of a text

texts: I am a highschool student working on a science fair project in which me and a friend plan to use a neural network for a classification problem. In this case the thing classified will be text and the labels will be the publication/creation date. Since our project requires a significant amount of data we were planning to use the Gutenberg project. However we came upon the issue that the creation date was not in each file but rather on a website associated with the name (so we have to go to a website which ends with the text's file name). We are wondering what the best approach would be to get our data - should we somehow get the date using the website (even though they say they will ban ip's of scrapers), find a new dataset, or build some sort of tool which will search the internet for that text's creation date. Thanks and if you have any suggestions please feel free to comment. If our question should be further expanded upon please clarify. Thanks again.


texts: If we only have the rating scores of items provided by the users, how do we use matrix factorization to build a recommender system model?

texts: If we only have the rating scores of items provided by the users, how do we use matrix factorization (MF), factorization machine (FM), and deep learning (DL) to build a recommender system model?


texts: sending files through a network with python

texts:
I'm building a multi-agent system  with python, the first agent built a model and saved it in yaml file, it saved also the weights in hdf5 file, then it must send this 2 files to the second agent.
the files are saved in my pc, and the 2 agents also are in my pc.
how could the first agent send the 2 files to the second one?
thank you.


texts: Extracting document templates from similar documents

texts:
Using very basic techniques (zone segmentation + OPTICS) I was able to organize a set of around 10^4 business documents (invoices, receipts) into hierarchy of clusters of documents of similar layout. Now, for each cluster, I would like to extract a template. A template consists of:

labels: text boxes whose position and content (text) remains fixed
inputs fields: text boxes whose position is fixed but content varies.

I have already OCRed my documents. As a result for each document I already have list of boxes with their position, size and content.

What would be the best approach to find labels and input fields for each cluster?

First thing that comes to my mind is to group boxes by their position, size and content (with some tolerance for position shifting and small spellig errors coming from OCR).


texts: How to generate a normal distribution with variance $\sigma^2= 0.08$

texts: I have a problem to solve.  I'm lost at generating a random data set for binary classification where each class correspond to the $2D$ region depicted in the figure below. The random data I must generate should have normal distribution with variance $\sigma^2 = 0.08$. I will use $200$ points ($100$ in each region) to train my nural network and report the results in terms of the loss function and the training epochs. New guy here Any tips will be welcomed!!


texts: Long sequence prediction with model trained on short sequence

texts:
I'll start with a specific example.

I would like to train model which predict vector of [0-1]. Values close to 1 on specific range indicates that in those timesteps is particular activation word (audio).

I wonder if I train model on, for instance, maximum 10 second recordings, then if I give much longer audio spectrogram to predict wake-word occurences (minimum few minutes long), then would I get sensible results? Does LSTM/GRU network remember some maximum range of timesteps, depending of some parameters? Or should I slice my data for prediction to smaller parts, close to 10 seconds, as length of training data? What's approach to predict long sequences from model trained on small sequences?


texts: How to add a new category to a existing trained deep learning model?

texts: i have trained my deep learning model initially with 5 classes now i want to add another class without training the whole model over again for those 5 classes. How can I do that?


texts: Understanding neural network probability

texts:
I'm training a CNN model with two classes to predict. I know it gives me a probability for one class and for the other one, and I also know I can get the predicted label, but I don't the results given. Isn't the sum of the output for each evaluated input supposed to be equal 1.0? For instance:

[[0.2858745  0.85059494]
 [0.2858745  0.85059494]
 [0.6040499  0.5927084 ]
 [0.8403308  0.291448  ]
 [0.04195209 0.95504093]
 [0.79433376 0.21279709]
 [0.79433376 0.21279709]
 [0.01326967 0.9891382 ]
 [0.0153821  0.9867737 ]
 [0.79433376 0.21279709]
 [0.01617167 0.98520505]
 [0.01351487 0.98596036]
 [0.01473185 0.9846144 ]
 [0.00896762 0.9899838 ]
 [0.00936404 0.9893628 ]]

Is there something I didn't get?

Code:

model_05_01 = Sequential()
model_05_01.add(Conv1D(filters=16, kernel_size=12,
                 input_shape=(x_train.shape[1], 1)))
model_05_01.add(MaxPooling1D(pool_size=4))

model_05_01.add(Conv1D(filters=32, kernel_size=12))
model_05_01.add(MaxPooling1D(pool_size=4))

model_05_01.add(Conv1D(filters=16, kernel_size=12))
model_05_01.add(MaxPooling1D(pool_size=4))

model_05_01.add(Flatten())

model_05_01.add(Dense(16, activation='relu'))
model_05_01.add(Dense(2, activation='sigmoid'))

model_05_01.compile(loss='logcosh', optimizer='adam',
              metrics=['accuracy'])


texts:
I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my experience with ML/data science is through Matlab.

Does anybody have insight as to skills that employers will expect me to add before I get a job? I've been looking at postings and it seems like most of the things that I'm not as sharp with are: SQL, Hadoop, Python, and R.

Are Python and R both necessary, or would that be redundant?
Is SQL and Hadoop something employers will overlook? Because quite frankly I find learning both to be unbearably boring.
Lastly, am I missing anything from this list of skills?


texts:
I have trained a model (Random Forest) and now I want to use it to predict for a certain data on a particular day. I have a categorical column where there are some values(say a,b,c,d,e) over a period. Now on a particular day, only some of those values are there (say b,d). Now while making them to onehot encoding, I am using LabelEncoder and the Onehot encoder. But if I give that column for label encoding, it is labelling only 'b' and 'd' (say 1 & 2) and the one hot vector length is 2. What I need is say the actual model labelled (a,b,c,d,e) as (1,2,3,4,5), now i need (b,d) to be labelled as (2,4) and the one hot vector to be of size 5.
What I am doing is saving the label encoder used in the training and using that one to label encode the column on that day. But I am not getting proper results, am I doing it the right way ?
I have given the length of onehot_train as n_values for the one day data. I have used sklearn label encoder and onehot encoder.
That is my main question and other one is, suppose I see new category which I haven't seen it during training, how to proceed with that, should I consider all the new categories as a 'unknown' category and encode all of them as same onehot or is there any better method?
Thank you for any suggestions.

def get_onehot(arr):
    label_enc = LabelEncoder()
    onehot_enc = OneHotEncoder(sparse=False)
    int_enc = label_enc.fit_transform(arr)
    int_enc = int_enc.reshape(len(int_enc),1)
    onehot = onehot_enc.fit_transform(int_enc)
    return onehot_train,label_enc_train

def get_onehot_per_day(arr_perday, label_enc_train, length_onehot_train):
    onehot_enc = OneHotEncoder(sparse=False,n_values=length_onehot_train)
    int_enc = label_enc_train.transform(arr)
    int_enc = int_enc.reshape(len(int_enc),1)
    onehot = onehot_enc.fit_transform(int_enc)
    return onehot_per_day


texts: Is my LSTM model overfitting or underfitting?

texts:
I am currently working on a project to classify comments text into 11 different topics, using a Bidirectional LSTM model. However, the loss curves confuse me as there is a deviation of the training and validation loss, a sign of overfitting when I tried the 100 neuron model, yet when I simplify the model to 30 neurons overall loss increases. Loss functions are as shown below:

The model was trained using 1727 comments, and the pre-trained GloVe 100-d embeddings. The max length of each comment is set to be 120. The validation dataset has 432 samples.
Code is as follows:
def bilstm():
    embed_dim = EMBED_DIM #EMBED_DIM = 100
    inp = Input(shape=(MAX_LEN, )) #MAX_LEN = 120
    x = Embedding(len(word_index)+1, #word_index being a dictionary of 5434 words
                  EMBED_DIM,
                  weights=[embedding_matrix], #embedding_matrix dimensions of 5434 x 100
                  input_length=MAX_LEN,
                  trainable=False)(inp)
    x = SpatialDropout1D(0.3)(x)

    x = Bidirectional(LSTM(10,
                           dropout=0.3,
                           recurrent_dropout=0.3,
                           return_sequences=True))(x)
    avg_pool = GlobalAveragePooling1D()(x)
    max_pool = GlobalMaxPooling1D()(x)
    conc = concatenate([avg_pool, max_pool])
    x = Dropout(0.5)(conc)

    outp = Dense(11, activation="softmax")(x)

    model = Model(inputs=inp, outputs=outp)
    model.compile(loss='categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])
    return model

bilstm_model = bilstm()

bilstm_history = bilstm_model.fit(x=train_seq,
                             y=y_train,
                             validation_data=(val_seq,y_val),
                             epochs=100,
                             batch_size=32,
                             verbose=1)

What changes can be made to my model to:

Improve accuracy/Minimize loss overall, given that my dataset is relatively small
Prevent potential overfitting, that is, to have the train and val curves move in tandem/not deviate


texts:
I have a dataset of customers:

age  height  weight eye_colour
30    174    74       Nan
20    191    71       Nan
28    165    56       Brown
...

I would like to identify similarities between customers where the eye_colour is Nan in order to help identify the cause of this missing value.

Is there any method to help identify similarities?


texts: Unidentified Image error while loading images on Google Colab

texts:
I'm writing a code to make movie genre classification with movie posters.
I opened a github repository where I put all the posters and cloned it on Colab. Until here, eveything works fine.
When I'm trying to load the pictures from the path in order to put them in an array , I get this error :
UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7f27dc3a1468>
When I tried with anoter dataset, it worked fine, but for this one , it doesn't work.
Here is my code


texts: When to use Gradient boosting over stochastic gradient boosting

texts:
Gradient boosting works on the Gradient Descent concept and it's one of the ensemble methods. It has a regularization parameter to select subsamples, which is called stochastic gradient boosting.
Gradient boosting uses many steps if the learning rate is low, to find the minimal. If the learning rate is high, it may skip the local minima. To avoid this, Stochastic gradient boosting used.
My doubt is when to use Gradient boosting and when to use stochastic boosting


texts: Is Difference Transformation required for LSTM?

texts:
I am Working on univariate multiple-step LSTM sequence prediction. My LSTM model is failing to give a good prediction on My Data. From some online blogs, I saw that Difference Transform may reduce Data Complexity.
Difference Transformation is   Preferred on the Arima like-model. But I never saw such an operation
done on LSTM.

what you people think, is differencing required for LSTM?


texts:
I have a classification problem. I want to reduce number of features to 4 (I have 30). I'm wondering why I get better result in classification when I use correlation based feature selection(cfs) first and then employ pca in comparison with just employing pca (the latter one is worse than the first one). It also should be mentioned that data loss in the second approach (just pca) 0.2-variance cover:0.8- and in the first one is 0.4 -variance coverd: 0.6!

Thank you in advance


texts:
I am having a hard time understanding why my loss is constantly a zero when using DQN.
I'm trying to use the gym environment to play the game CartPole-V0.

In the code below,
r_batch indicates rewards sampled from the replay buffer, and similarly s_batch, ns_batch, and dones_batch indicate the sampled state, next states, and if the "game" was done. Each time, I sample 4 samples from the buffer.

# This is yi
target_q = r_batch + self.gamma * np.amax(self.get_target_value(ns_batch), axis=1) * (1 - done_batch)
# this is y
target_f = self.model.predict(s_batch)
# this is the gradient descent computation
losses = self.model.train_on_batch(s_batch, target_f)

The values of target_q are:

[ 0.42824322  0.01458293  1.         -0.29854858]

The values of target_f are:

[[0.11004215 1.0435755 ]
 [0.20311067 2.1085744 ]
 [0.413234   4.376865  ]
 [0.24785805 2.6716242 ]]

I'm missing something here, and I don't know what..


texts: Is there such thing as linear and non-linear data?

texts:
While doing machine learning projects we've heard that logistic regression works well with "Linear data" and decision tree works well with "non-linear data"
However concept of linear and non-linear data does not make sense. To me only linearly separable data and non-linearly separable data makes sense to me, it only makes sense to say logistic regression works well with "Linearly separable data" since it is a linear function. In mathematics linear functions are polynomials with degree one and all other functions that are not linear are considered non-linear function.
What exactly is linear data and non-linear data?


texts:
"I want to go swimming next Tuesday"
I want to machine to learn the date I want to go swimming. Is there any approaches or libraries that can

Extract "next Tuesday"
And calculate the exact date?


texts: Understanding, visualizing and interpreting CNN activations

texts:
I am working with the first layer of a CNN and trying to understand how to interpret the activation output. My CNN takes input from 3 channels (RBG picture) and the first layer is Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False).
From this I understand that every input picture will be convolved with weights of size 64x3x7x7 in order to derive the output.
After this is done I would like to visualise the output (the activations of the layer) but I have the following questions:

Should the activations be normalised for the visualisation and if yes how? My question essentially boils down to "Is activation relative"?
For example if for channel 1 the max value of the activation pixels is 4 while for channel 5 the max value of the activation pixels is 10 should I normalize both activations based on their respective max(4 and 10) or the total max (10). And how should I interpret this?

Activations that do not show pixel variability can be interpreted as "The pattern that the corresponding weight kernel is looking for is not present in the image?"


texts: Policy network AlphaGo and transferring to other domains

texts:
This is not covered very well in their AlphaGo paper but I assume that their policy network has a softmax output layer with a node for all the positions on the board, including illegal ones (the one with a stone on the board already). Maybe the illegal ones are basically zero already, I'm uncertain, but I assume they manually put them to zero and then normalize the probabilities. Please correct me if I'm wrong.

To dip my toes into using neural networks for game AI I decided to work on a Backgammon player. I have only implemented a 1D convolutional value network on the board and merged the convolutional layers with the bar and the bearing states. I let random players play a bunch of games, train the model on those game states and let the model play against itself a number of times, tune the model further and iterate this step a number of times. The results are promising so I was looking to add a tree search algorithm similar to AlphaGo.

This is where my question comes from. I would like to use a policy network to guide the exploration through the tree, but I don't know how to represent it. If you don't take the current state in mind, you would have a huge amount of potential states in your last layer of which more than 99.99% would be illegal. If you do keep the current state in mind I don't see how I could train a network on this. Is there any way of building a policy network for Backgammon?

An alternative method would be to use the value network on the potential states that you are able to go to (the network outputs a probability of winning for the active player) and transform those probabilities into a new distribution for exploration, taking in mind exploration versus exploitation.

EDIT: My current representation of actions is the state it would transfer into by doing that action, which means the number of actions over all possible games is enormous.


texts: An ambiguity in SVM equations about misclassified data

texts:
I have encountered an ambiguity in SVM equations.
As is stated in Chris Bishop's machine learning book, the optimization goal in SVM is to maximize this function:

$$C\sum\limits_{n = 1}^N {{\xi _n}}  + {1 \over 2}{\left\| w \right\|^2}$$

Subject to this constraints(*):

$${\xi _n} \ge 0$$
$${t_n}y({x_n}) \ge 1 - {\xi _n}$$

where:

$$y({x_n}) = {w^T}{x_n} + b$$

so the corresponding Lagrangian function for this problem is:

$$L(w,b,a) = C\sum\limits_{n = 1}^N {{\xi _n}}  + {1 \over 2}{\left\| w \right\|^2} - \sum\limits_{n = 1}^N {{a_n}\{ {t_n}y({x_n}) - 1 + {\xi _n}\}  - } \sum\limits_{n = 1}^N {{\mu _n}{\xi _n}} $$

and the corresponding KKT conditions are given by (**):

$${a _n} \ge 0$$

$${{t_n}y({x_n}) - 1 + {\xi _n}} \ge 0$$

$${a_n}({t_n}y({x_n}) - 1 + {\xi _n}) = 0$$

$${\xi _n} \ge 0$$

$${\mu _n} \ge 0$$

$${\mu _n}{\xi _n} = 0$$

And if we set

$${{\partial L} \over {\partial {\xi _n}}} = 0$$

we get (***)

$${a_n} = C - {\mu _n}$$

As we know, that subset of data points that have

$${a_n} = 0$$

are not support vectors. But for this data points we have (from ***):

$${\mu_n} = C$$

and therefore (from **)

$${\xi _n} = 0$$

So here lies the problem. If a data point from this subset is in the wrong side of the decision boundary, then

$${t_n}y({x_n}) \le 0$$

and we will have (from *)
$${\xi _n} \ge 1$$

which is in an obvious conflict with

$${\xi _n} = 0$$


texts: Error on multitask neural nets where all outputs not observed for every example

texts:
Let's say I have 2 datasets, each from a set of experiments.  Dataset A measures a set of properties X for set S, while dataset B measures properties Y for set T.  X and Y are highly correlated, and S and T have (not perfect) overlap.  To give an example, I might have

A   |   ID   |   pKa    |  log P
    |  cid12 |   3.51   |   1.2
    |  cid51 |   2.32   |   .9

B   |   ID   | Pol Srf A|  cLogP
    |  cid12 |   48.5   |   1.5
    |  cid88 |   61.2   |   .9

I would like to use the shared information here (since all of these are physical properties) in a multitask neural net setup.  For examples in both S and T, I have a target vector as long as X union Y.  But for members exclusive to S or T, I am missing values.

In the examples, I have

 C   |   ID   |   pKa    |  log P  | Pol Srf A|  cLogP
     |  cid12 |   3.51   |   1.2   |   48.5   |   1.5
     |  cid51 |   2.32   |   .9    |    na    |   na
     |  cid88 |    na    |   na    |   61.2   |   .9

When I train, do I simply zero fill these outputs, then not backpropagate error coming from these nodes?  So for  cid12, error from all nodes backpropagates, but for cid51, only error from pKa and logP?

If this is the case, is that a standard procedure implemented in most NN libraries (Theano, for example) and what is that called?


texts: Why can we not split train test data with 0.01 as parameter or 99% training data

texts:
Most of the blogs mention about a good thumb rule to be 80-20 split for the train and test respectively.

My special case is a time series dataset and it is for the stock prices, which IMO is very time sensitive.

Question? Why can we not have a 99-1 train test split, for the model to learn all the information and time trends. Since my prediction will be in the future I will be ever increasing my test data set. I am using a neural network (rnn-lstm) for my prediction.

I am aware about cross_validation, froward_chaining which are better ways to train a time series dataset.


texts: Is there any TF implementation of the Original BERT other than Google and HuggingFace

texts:
Trying to find any Tensorflow/Keras implementation of the original BERT model trained using MLM/NSP. The official google and HuggingFace implementations are very complex and has so much of added functionalities. But I want to learn and implement BERT for just learning its working.
Any leads will be helpful?


texts: Is the magnitude of the gradient a weakness of Gradient Descent?

texts:
The formula for Gradient Descent is as follows:
$$
\mathbf{w} := \mathbf{w} - \alpha\; \triangledown C
$$

The gradient itself points in the direction of steepest ascent, therefore it is logical to go in the opposite direction by subtracting it. But besides the direction, the gradient also has a magnitude which actually doesn't say anything about the path to the optimum.

My question is, is this considered a weakness of gradient descent and the reason for the learning rate?


texts: Clustering and producing final results to find next best customer to target(Ranked)

texts:
I have a problem where I need to cluster customer data that has all possible attributes to identify the next potential customer who can succeed the last customer in terms of buying a certain product. My data frame has about 5000 rows and 51 features from which I need to derive this result. I understand that first I need to perform clustering in order to form the right groups but what next? I have already done the following steps :

 EDA
 Correlation analysis
 Enoding of categorical features
 Scaling of the data basis standard scalar
 PCA - Achieved a final set of 21 components retained within 90% variance

First, how do I used the final components in the form of a dataset into a clustering algorithm? Second, What would be the best-suited clustering method for such a problem? Hierarchical- agglomerative?

After clustering, I need to build a model in such a manner that the final output should give me customer ID and the next best 5 customers ranked in an order basis the strongest attributes combination. For example: Spend, Gender, Salary, Area they live in, etc

This is an unsupervised problem but eventually, I need to split the dataset into a test data and train data with a target variable "Important Customers" that has a flag of 0 and 1 that defines my potential important target customers. The model should be able to give me a list of customers with their 5 potential successors in a ranked manner(highest to lowest)


texts: How to check two images(one is original image and other one captured by mobile) similarity using deep learning?

texts: My problem statement is - In my project, original image of product is stored in database. Now whenever any person uploading that product's camera pic(for internal audit process) then I need to verify whether this uploaded image is similar of original image or not. So can anyone help me on this, how can I achieve this verification automated using deep learning or tensor-flow ?


texts: Training set Distribution and Activation function/Loss function correlation

texts:
How should the probability distribution of the training set influence the choice of the activation function / loss function?
For instance if I have a Multinoulli distribution, which activation function should I choose? And why?
I can't get this correlation between the probability distribution of the training set and the choice of the activation function / loss function.


texts: How do I use neural network to predict variables in linear equations?

texts:
I am going to calibrate two monitors. Two variables "Brightness" and "Contrast" can be adjusted to obtain the monitor's color properties (Its pixel values).

First, I use some calibration equipment to measure the first monitor's color space property. Then, I use a regression method to obtain a straight line representing the gray level change of that panel. Hence, I find the slope of that line representing the contrast and the y-intercept representing the brightness correlated to that monitor.

My question is how to apply my prior knowledge to adjust another monitor's brightness and contrast as same as my first monitor?

Assume that I also use calibration equipment to measure the second monitor's color property.

Can I use neural network to obtain the true value of contrast and brightness of another monitor?


texts: Is there any way of ordering/sorting vectors?

texts:
I am working on KD-Tree for nearest neighbor algorithm, where at each level of the tree we arbitrarily choose a dimension to cut upon and sort the points based on that chosen dimension value, after which we can choose mid-point to split upon such that dividing the points from that middle point will give us a balanced tree (has same number of points in either of the subtrees). At each level a single dimension is chosen and process is repeated.

For example: for data having dimensions (x,y,z) let's say first level of the tree 'X' is chosen at cut dimension, 'Y' at second level, 'Z' at third and then repeated again with 'X' again at fourth level and so on. So for first level all points are sorted according to dimension 'X' and split into left and right subtree based on the middle value of the sorted data.

I am sure you got the gist of it. Otherwise please read more here. KD-Tree (Wiki)

Now what I want to try something where rather than using each dimension at different level, I use all at the same level. i.e. I take points p(x,y,z) as vectors. But as you can guess if I do this, I have no way of ordering (not any I know of) the vectors (points) such that I can find a vector to divide upon hence still keeping my tree somewhat balanced.

PS: I am not much concerned about if it is actually a better or worse way of doing thing. Right now it's more about if it is possible to do it this way and then find out why it is better or worse.

Please ask if I should provide any more information or if I mentioned something wrong. I appreciate your help.

Please add more tags if appropriate, I couldn't find any.

Edit: Just to make it clear. Basically I want to be able to have an ordering which won't destroy the order based on distances. i.e. if a point/vector A has less distance from another point P, it comes first  in the order than some farther point, let's say B.
My end goal is to use this to try and create some kind of nearest neighbor algorithm.


texts: Deep learning model gives random results

texts:
First I am new to machine learning if it is an obvious question, I am sorry.

dataset_coefficients = loadtxt(
'in.csv', delimiter=',')
dataset_answers = loadtxt(
    'out.csv')
X = dataset_coefficients[:, 0:4]
y = dataset_answers

model = Sequential()
model.add(Dense(32, input_dim=4, activation='relu'))
model.add(Dense(16, activation='sigmoid'))
model.add(Dense(8, activation='sigmoid'))
model.add(Dense(4, activation='sigmoid'))
model.add(Dense(2, activation='sigmoid'))
model.add(Dense(1, activation='softmax'))

model.compile(loss="mean_squared_error",
              optimizer='adam', metrics=['accuracy'])

model.fit(X, y, epochs=250, batch_size=100, verbose=0)
_, accuracy = model.evaluate(X, y, verbose=0)
print('Accuracy: {}%'.format(accuracy * 100))

in.csv looks like this

     1,       -21,       147,      -343
     1,        19,       115,       225
     1,         1,       -64,       -64
     1,        30,       300,      1000
     1,        16,        64,         0
     1,         3,       -81,      -243
     1,         3,         3,         1
     1,       -13,        16,       192

out.csv

     1
     2
     3
     1
     2
     3
     1
     2

When I test the accuracy with the data which was previously used for training

Output is

Accuracy: 33.33333432674408%

No matter how big my dataset is, the result has the exact same accuracy rate of 33% which seems like the prediction is literally random. What am I doing wrong? Thanks


texts: Understanding text conversion into SVM input

texts: In Support Vector Machines, when used for sentiment analysis, text gets converted into a set of data points. How does this happen, usually?


texts: Characterisitcs of the data set for a binary classification problem

texts: I want to build a classifier for my problem statement and for that I don't have data. So while doing data acquisition, what should be the minimum sample size? And would it be a good practice if I label each observation myself to build a valid data set? (I cannot automate the process of labeling observation to each class while doing data acquisition and manually doing that takes up lot of time)


texts:
I am writing a TensorFlow program which is trying to categorize a heavily skewed dataset between two categories. One category is represented at 30x the rate of the other.

Category 0: 800
Category 1: 30000

The label for each record is a one-hot vector of length 2, representing the two categories. Using a cross entropy cost equation doesn't work very well since a very high accuracy can be attained just by rating everything as category 1.

I then made a balanced dataset that has the following data:

Category 0: 800
Category 1: 800

This dataset has about 67% accuracy using softmax, but there is a real-world cost when falsely categorizing a category 1 item as category 0, which is blown way up when mistakenly categorizing many more items as category 0 mistakenly.

I wanted to fix this by making the cost equation penalize false categorization into category 0 more than other penalties. However, my cost equation is not producing any different results than the original cross entropy equation. I think I may have programmed it in wrong, but I am unsure where the mistake is.

In theory, the skewed equation will add 1 to the normal cost when the label for a record is [0,1] and the predicted value is [1,0], which represents a miscategorization of a category 1 item into category 0.

The skewed cost equation is in the condition where FLAGS.cost == 'skewed':

x = tf.placeholder(tf.float32, shape=[None, data_length])

W = tf.Variable(tf.zeros([data_length, 2]))
b = tf.Variable(tf.zeros([2]))

y = tf.nn.softmax(tf.matmul(x, W) + b)

y_ = tf.placeholder(tf.float32, shape=[None, 2])

if FLAGS.cost == 'cross_entropy':
    cost = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y,1e-10,1.0)))
elif FLAGS.cost == 'skewed':
    positive_cutoff = tf.constant([0.5, 0.5])
    desired_tensor = tf.constant([False, True])
    true_tensor = tf.fill(tf.pack([2, tf.shape(y)[0], 2]), True)
    cost = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y,1e-10,1.0)))+tf.to_float(
        tf.reduce_all(
            tf.logical_and(
                tf.pack([
                    tf.logical_and(tf.greater_equal(y_, positive_cutoff), desired_tensor),
                    tf.logical_and(tf.less_equal(y, positive_cutoff), desired_tensor)
                ]),#shape [2,x,2]
                true_tensor#shape [2,x,2]
            )#shape [2,x,2]
        )#shape [1]
    )

train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


texts: Get right shape of numpy array - (4112, 20, 28, 28)

texts:
I'm trying to create an numpy array that resembles = (flattened images, number of examples per class, width, height)

This is the original code:

train_dataset = np.zeros([no_of_classes, num_examples, img_height, img_width], dtype=np.float32)

for label, name in enumerate(train_classes):
    alphabet, character, rotation = name.split('/')
    rotation = float(rotation[3:])
    img_dir = os.path.join(root_dir, 'data', alphabet, character)
    img_files = sorted(glob.glob(os.path.join(img_dir, '*.png')))

    for index, img_file in enumerate(img_files):
        values = 1. - np.array(Image.open(img_file).rotate(rotation).resize((img_width, img_height)), np.float32, copy=False)
        train_dataset[label, index] = values

train_dataset.shape

(4112, 20, 28, 28)

I wish to achieve the same result(the shape, not the numbers per se) with 2 examples per class, given that i have 5 classes.

I've tried:

data = [np.zeros((2000)), np.zeros((1)), np.zeros((40)), np.zeros((50))]

test = [np.array(x).flatten() for x in a]
test1 = np.array(test)
test1.shape

(5, 2000)

How can i do that?


texts: Is there a way of performing stratified cross validation using xgboost module in python?

texts:
I am training and predicting on the same data-set, but I want to perform 10-fold cross-validation and predict on the left out fold and thus predict on the whole data set. How can I do this?

The libraries which I am using are:

from sklearn import cross_validation
import xgboost as xgb


texts: How can I find company descriptions for a long list of companies?

texts:
I'm going to train an ml algorithm to qualify potential sales leads based upon company descriptions. To do this, I need to find the company descriptions programatically.

E.g. given a long list of company names, how can I find descriptions for these companies. Here are my current techniques, which works ok, but not great:

Using Google Search API to and fetch summary from the results page (Google normally gives the company website as the first result when searching by company name)
Fetch all data from the company's website (includes an about-page and a lot of other less relevant data)
Using the FullContact Company API to get any descriptions the company has on its social profiles (works great if the company has an online precense)


texts: Question about reshaping array size for KNN Classifiers

texts:
I keep trying to run a new set of data through my KNN Classifier but would recieve the message:
 ValueError: query data dimension must match training data dimension

It then used:
x_new = pd.read_csv('NewFeaturePractice.csv' , names = attributes)
x_new = x_new.values.reshape(52,84)

(which is the dimensions of the training data) but would then receive:
ValueError: cannot reshape array of size 672 into shape (52,84)

The second data set doesn't have the same amount of rows as the first meaning that even if I tried reshaping the array I would have several empty spaces. How can I run the code so that no matter the size of my new data set, I can still get results?


texts:
I have some data which I know is well approximated as a trig function, and I can fit it with scipy.optimize.curve_fit as follows:

from __future__import division
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit

#Load the data
data = np.load('example_data.npy')

x = data[:,0]
y = data[:,1]

#Define the parametric trig function:

def trig_function(t,A,B,omega,offset):
return A*np.sin(omega*t) + B*np.cos(omega*t) + offset

#Define fitting procedure
def fit(x,y):

    sigma = np.ones(len(t))
    sigma[[0, -1]] = 1e-1 #weight the end points

    func = trig_function #define the fitting function

    #Some guesses for initial parameter values
    dA = (y.max() + y.min()) / 2
    y_shifted = y - offset
    omega = np.pi * np.sum(y_shifted[:-1] * y_shifted[1:] < 0) / (t.max() - t.min())
    p0 = (dA,dA, omega,dA)

    #Do the fit
    popt, pcov = curve_fit(func, x,y,p0=p0,sigma=sigma)

    #return fitted data
    return func(x, *popt)

#Define plotting environment
fig = plt.figure(figsize=(24,10))
ax1 = plt.subplot2grid((2,1), (0,0))
ax2 = plt.subplot2grid((2,1), (1,0),sharex=ax1)

#get fit data
y_approx = fit(x,y)

#Plot both solutions and relative error
ax1.plot(x,y)
ax1.plot(x,y_approx)
dy = (y_approx - y)/y
ax2.plot(x,dy)

This works pretty well:

However there is clearly still some feature not being accounted for by the model, especially close to the minima, as evidenced by the peaks in the relative error.

Can anyone advise how the model (i.e. the parametric trig function) could be further developed to properly account for the behaviour?

Thanks


texts:
Right now I'm studying Convolutional Neural Networks.
Why must a CNN have a fixed input size?
I know that it is possible to overcome this problem (with fully convolutional neural networks etc...), and I also know that it is due to the fully connected layers placed at the end of the network.
But why? I can not understand what the presence of the fully connected layers implies and why we are forced to have a fixed input size.


texts: can I use t-sne or PCA to reduce number of classes?

texts: I wanted to know if I can use t-sne or PCA to reduce the number of classes depending on the similarity between them. For example, if I have 100  classes of 100 different animals and would like to put all the cats in a group and all the dogs in a group etc. (to get few groups of these 100 classes).


texts: Information retrieval / slot filling / NLP

texts:
Excuse if this has been answered before.

I need to extract features and parse from a piece of text and run some analysis. For e.g. "Plot the past 5-year sales of Apple" should give me the following

Information:

Company: Apple
Item: Sales
Period: past 5 years
Action: Plot

What deep learning techniques / algorithms should I be looking to use?

Any pointers are highly appreciated.


texts: Reinforcement Learning techniques to use if there is a direct connecction between (state, action) and reward

texts:
I want to build a model. The input state is a combination of different numbers in sequence (Their orders matter to the final result). The effectiveness of such number combination can be computed into a real numbers through continuous and differentiable math formulas. This result is also used to compute a reward for each action.

My actions for each state (number combination) is to select a specific number at a position and change that number to any element from a finite list of numbers.

Are there any suggestions for a good reinforcement technique to solve this problem?

Any bits of help is appreciated!


texts:
I have a set of user transactions with a service and I suspect that there is a strong correlation between how users rate the service and how likely they are to use the service again.

I have chosen to represent user ratings - for each user in the dataset - as a number [1-5] which is the average of all ratings the given user has given in the past. There is more to it in regards to how to represent trends etc, but this is not in context for the problem at hand.

My issue is that there is a high number of users that have never left any ratings at all and I am not sure how to deal with these users, in terms of finding the right value for the rating-related features for them.

What I have tried so far is to represent ratings in a different way, shown below:

UID    rating_high    rating_medium    rating_low
  A              2                0             1
  B              0                0             0

where each of the predictors represent the count of ratings - in each rating category - by each of the users in the dataset.

In the above case, user A has left a good rating in 2 occasions, a low rating in one occasion and has never left a 'medium' rating. User B has never left any ratings for the service at all, therefore, he is assigned a count of 0 for all 'rating categories'.

on the 1-5 rating scale, I class all ratings below 3 as rating_low, all those equal to 3 as rating_medium and the ones above 3 as rating_high.

I have not managed to find any other way to represent this data, but I would very much rather not drop this part of the dataset as I believe it caries valuable semantics for my problem.

Any ideas on how to best deal with this issue are most appreciated!


texts: Imbalanced dataset - Undersampling & multiple classifiers

texts:
Let's suppose that my dataset in a classification problem looks like that:

class A: 50000 observations
class B: 2000 observations
class C: 800 observations
class D: 200 observations

These are some ways which I considered to deal with this imbalanced dataset:

I reject straight away oversampling because it usually makes the model overfit (in the minority classes) by a lot.
Secondly, if I run the classifier with the data like that then it will be overclassifying documents in class A so I reject this method too.
Another approach is to do undersampling and reduce class A to let's say 4000 documents (where I tested it and it gives the best results so far).
However, in this way I am losing quite a lot of information. So I am wondering if building multiple classifiers with 4000 documents each for class A (different at each time) is a better solution (although I think that this approach resembles quite a lot the oversampling approach which I rejected).

What do you think of method (4) comparing to method (3)?


texts: How effective would a data-driven vaccination program be over a simple rules based approach

texts:
The rollout of vaccination programs tends to be based on a given set of rules (advice) devised by government UK example. Such a rule-set are generally limited in complexity in order to be consistently applied by humans administering the roll-out.
There is a strong association between age and risk nature but it appears below the age of 60 that other factors such as sex/deprivation/ethenicity predominate. There is also mounting evidence towards mental disorders holding a significant risk factor risk factors and mental disorders
Is there any openly available papers or modelling showing a data-driven prioritization of vacination (for example using machine-intelligence), especially in the scenario of limited vaccine availability, would be more effective than simply using age alone as the predicate.


texts: Setting activation function to a leaky relu in a Sequential model

texts:
I'm doing a beginner's TensorFlow course, we are given a mini-project about predicting the MNIST data set (hand written digits) and we have to finish the code such that we get a 99% accuracy (measured as 0.01 loss) in less than 10 epochs.
In principle I am getting the accuracy, but the loss only reaches <0.01 at the 10th epoch (hence assignment is counted as failed).
As per instructions, I'm not allowed to change the model.compile arguments, so I decided I can try to change the activation function to a leaky relu, using the code I was given. It is not as straightforward as it seems and everything I found online does not seem to work. Most suggestions are in the model.add() format, which I can't figure out how to incorporate/substitute in the code they provided us, without changing it too much (everything failed).
The current code is given below:

  model = tf.keras.models.Sequential([
          keras.layers.Flatten(input_shape=(28,28)),
          keras.layers.Dense(128,activation=tf.nn.relu),
          keras.layers.Dense(10,activation=tf.nn.softmax)
      ])

Any help would be appreciated!


texts: Interpretation for test score , training score and validation score in machine learning?

texts:
Interpretation for test score , training score and validation score ? what they actually tell us?

What's an acceptable difference between cross test score , validation score and test score?

If difference between test score and training score is small mean it is a good model/fit?

overfitting and under fitting on the basis of test score , training score and validation score?

whether any of these score or difference among them tells us , if we need more data(observations)


texts: Cross-validated average: metrics mean or ensembling probabilities?

texts:
Let's say I have 5 models cross-validated via leave-one-out strategy. I have the predictions and scores of each model.
Now, it's time to calculate the average for the set of 5 models - am I supposed to:

add up the 5 losses and divide them by 5?
Or average their probabilities for each prediction and use the average probability to calculate new metrics like an ensemble/ forest?


texts: How to determine if problem is solvable by neural network?

texts:
I want to identify subtle patterns in images using a convolutional neural net. I have seen several examples where people gave up reasoning that the pattern is not dominant or consistent enough to be picked up by a neural network. Since tuning these networks is very time consuming I wonder: What are methods to validate that the problem is solvable/the pattern can be recognised (or at least give an indication of the same)?

Background: My problem

My goal is to identify subtle differences in products based on their images. I want to break down a product image into a large variety of describing attributes. Now I need to find out which product attributes are well suited for identification.

For example, I want to distinguish collar types in clothing images. Therefore I need to figure out if the human understanding of collar types (waterfall, V-shaped, round etc.) are distinct and consistent enough to be properly identified by a neural net with human level performance.

Of course images always differ a little and the solvability is clearly related to the data available. But before collecting and cleaning hundreds of thousands of images I would like to find out if results will be any good.


texts: AWS SageMaker Model as endpoint size limit

texts:
Is there a size limit imposed on models deployed on AWS SageMaker as endpoints? I first tried to deploy a simple TensorFlow/Keras Iris classification model by converting to protobuf, tarring the model, and deploying. The size of the tarred file was around 10KB, and I was able to deploy that successfully as an endpoint. However, I tried the same process with a Nasnet model where the size of the tarred file ended up being around 350MB, and I got the following error:

  The primary container for production variant AllTraffic did not pass
  the  ping health check. Please check CloudWatch logs for this
  endpoint.

Could it be because the model is too large to deploy? I tried increasing the instance type from 'ml.m4.xlarge' to a higher tier but that did not work either.


texts: Can anyone verify my NN diagram if it is properly drawn?

texts:
I am working on a Neural Network that can estimate building's carbon footprint based on the set of features and an image of urban surroundings (via CNN).

I have used Netron to visualize the network (up) but this image is not readable in a publication so I drawn one myself (down).

Can anyone comment if this is proper representation - especially the last part after the concatenation - I am not sure if there should be 2 or 3 dense layers?


texts: Do I need to train a separate DeepFake model for every input person?

texts:
I would like to create a deep fake model of a specific person (we will call him Steve). I would then like to be able to upload a video of any random person and swap their face with Steve's.

So far I have seen most deep fake systems require datasets of person A and B. Is there a way to only train on person B and have a general version of A so that I can allow anyone to be the input?


texts: Aggregating small values in a frequency bar plot

texts:
I have a pandas Series of sorted percentage values like this :

A -> 0.001
B -> 0.0012
C -> 0.0015
...
H -> 0.02
I -> 0.03
J -> 0.041
...
X -> 0.12
Y -> 0.31
Z -> 0.4

(I typed those by hand, I am not familiar with how to type those python outputs in DSSE, sorry! If anyone has a hint on that, it would be nice. If this pandas series would be named "content_distribution", content_distribution.X would return the float 0.12 .)

I want to obtain a Series which looks like this :

Other -> 0.07
I -> 0.03
J -> 0.041
...
X -> 0.12
Y -> 0.31
Z -> 0.4

because those small values are irrelevant to display in a plot that I want to make, and I want to keep the big values on display. Is there any pandas-ic or pythonic way to do that? I can write my own script, but anything I try out looks non-pandas-ic (even though it sort of works).

(I also don't know if "aggregating" is the correct pandas term for this, if anyone has a better one, he or she is welcome to edit my title.)


texts:
I'm trying to build a toy model which can identify a constant difference between two variables: (if variable1- variable2>10 then 1 else 0).

This should be a quite simple task for any regression model, but I want to solve it with NN. However all simple NN I built can not give me more than 51% accuracy.

Is this something that I do not understand?

My code:

seed = 7
np.random.seed(seed)

x1 = np.arange(50000)
x2 = x1+10+(0.5-np.random.rand(len(x1)))
X = np.column_stack((x1,x2))

Y = (x2-x1)>10

encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)

train_X, test_X, train_y, test_y = train_test_split(
    X, Y, train_size=0.9, random_state=0)

model = Sequential()
model.add(Dense(2, input_dim=2,activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1,  activation='sigmoid'))

model.compile(optimizer=RMSprop(lr=0.001),
  loss=binary_crossentropy, metrics=[binary_accuracy])

history = model.fit(train_X, train_y, epochs=1000, batch_size=10,
  validation_data=(test_X, test_y), verbose=1)


texts: Does Naive Bayes Classifier require assuming a specific distribution for training?

texts:
Since NB is a generative classifier, we assume that the data points are all generated from a distribution, right?

But since we can compute the MLE of p(x_i|y) by counting (MLE of p(x_i|y) = # of (x_i,y) / # of y), do we really need a specific distribution, if at all, to model the likelihood(p(x_i|y))?


texts: My Stacked LSTM seems to be doing worse than a shallower one

texts:
I started with a two layer LSTM (+ Dense Layers) and which was:
my_model.add(LSTM(units = 64,input_shape = (33,1), return_sequences=(True)))
my_model.add(LSTM(units = 128))

This gave me not great, but not terrible losses as shown

My guess was that while my model is still learning,it would work a lot better with more layers, but it only got worse with an extra layer.
I implemented the stacking like this:
my_model.add(LSTM(units = 64,input_shape = (33,1), return_sequences=(True)))
my_model.add(LSTM(units = 128, return_sequences=(True)))
my_model.add(LSTM(units = 128))

And the result I got is significantly worse:

This really makes no sense to me. My guess is that it could be a case of and undertrained model caused by not enough data.
Edit: The rest of my model (common for both cases)
my_model.add(Dense(100))
my_model.add(Dense(10))
my_model.add(Dense(1))
opt = tf.keras.optimizers.Adam(learning_rate=0.003)#changed from 0.001
my_model.compile(loss = 'mse',optimizer = opt, metrics = ['mean_squared_error'])
my_model.fit(X_train,y_train, batch_size = 512,epochs = 100,validation_data=(X_val,y_val),shuffle=True)


texts: Best method to deal with too many zeroes in regression problem?

texts:
I'm trying to solve a simple regression problem using TensorFlow and Pandas to see what's the expected conversion rate for a given product given how much I'm willing to pay for each visit (cost per click or cpc).

Each data point represents a single day and its given acquisition cost and conversion rate. I expect conversion rate to decrease exponentially as visits increase because my audience gets wider and not as focused anymore.

The data seems to agree with this:

The problem is that for the majority of data points, the conversion rate is zero (i.e. no products were sold) so any model is likely to perform well by merely guessing zero sales. For this particular subset (one of the best selling products) the ratio of zero-to-non-zero is roughly 4:1 but in the general data it's about 300:1.

The two best ways that I can think of dealing with this are

a) summarizing data by time periods large enough to contain at least one conversion in them (problem with this is that I lose data by having to settle for the average cost instead of the more detailed daily data)

b) splitting the problem into a classification problem (will I sell at least one unit) and a regression problem (given that I'm going to sell at least one unit, how many units can I expect to sell)

Do either of those sound like a smart idea? Are there any tried-and-true methods to solve this kind of problem?


texts: Convolutional network for classification, extremely sensitive to lighting

texts:
I trained a convolutional network to classify images of a mechanical component as good or defective. Though the test accuracy was high, I realized that the model performed poorly on images which had slightly different lighting.

The features that I'm trying to detect are subtle, and lighting seems to trip the model into calling a lot of false negatives. Most, if not all, of the training and test data came from images which were shot under diffused lighting. The new images that I tried the model on were shot with focused lights.

Hoping that histogram equalization(CLAHE) would help, I did equalization on
images before feeding it to the model, did this for training as well as test data. The problem then was that accuracy was high, but the model seemed to have learnt some other correlation between the images and labels. After equalization, everything in the image stands out, the defects become even more subtle and hard to detect even for human eye.

The model prediction changed even when the same component was shot with the same lighting but with the light at different orientations.
So my question is, what can I do, either at the data end or with the model, so that it becomes robust to changes in lighting(or same kind of lighting but changes to orientation of lighting)?


texts: Why does Q-learning use an actor model and critic model?

texts:
I'm currently reading Hands on Machine Learning with Scikit-Learn & Tensorflow, and I'm wondering why does Q-learning require an actor model and a critic model to learn?

On page 465, it states:

  As we will see, the training algorithm we will use requires two DQNs
  with the same archicture (but different parameters): one will be used
  to drive Ms. Pac-Man during training (the actor), and the other will
  watch the actor and learn from its trials and errors (the critic).

Is this a typical Q-learning implementation? If not, what is?


texts: How can I check if a bigger training data set would improve my accuracy of my scikit classifier?

texts: How can I check if a bigger training data set would improve my accuracy of my scikit classifier, is there a method or something?


texts: Fixing attention of CNNs for image classification

texts:
I'd like to fix attention of a CNN to a particular region of a given image.

For example, given 28x28 image, I'd like my model to only consider only the top-left 5x5 region.
What's the proper way of doing this, if possible?

A naive work-around I can think of is that, I could overwrite the unwanted regions with black-pixels. Not sure if this is a nice solution though.


texts:
I'm trying to apply emotion detection on audio. Basically, given an audio file, it must generate emotional labels for a segment of any durations, say every 5s chunks, like happy, sad, angry, etc.
Is there a working example with pretrained models available to test? I was looking for a sample on GitHub, but had problems running it. I need a relatively accurate model. Any working sample would be helpful.


texts:
I have very less training observations (15). I need to predict 6 months into the future. What forecasting model is best suited for this scenario? Here is how my dataset looks
Month        | Response Rate       |% Promoters          |% Detractors        |%Neutrals
2019-01-01     | 5%                 |60%                 |30%                | 10%
2019-02-01
.....
2020-07-01
I need to predict Response Rate, % Promoters, % Detractors and % Neutrals all of which are numeric variables.
I am new to this forum, so pardon me if I have done any mistake while framing the question.


texts:
create model

model=Sequential()
model.add(Dense(input_dim = 12, units = 10, activation='relu', kernel_initializer='uniform'))
model.add(Dense(units = 8, activation='relu', kernel_initializer='uniform'))
model.add(Dense(units = 1, activation='sigmoid', kernel_initializer='uniform'))

print('Training the model...')

Compile model

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

while running the model.fit()

Fit the model

model.fit(X_train,y_train,batch_size=32, epochs=10, verbose=1)

I am getting the Error

  ValueError: Error when checking input: expected dense_36_input to have shape (None, 12) but got array with shape (140, 2)


texts: Best MIMO prediction algorithm for categorical variables

texts:
I have researched machine learning for quite a while and would like to test out my knowledge. So I am trying to use it for lottery number prediction. The goal is not to have 100% correct prediction (which is of course impossible), but to perform better than pure random prediction.

Basically, I have the following data:

[
 [3,10,16,19,34,45],
 [7,14,15,20,28,41],
 [2,6,18,24,30,37],
...
]

As it can be seen each round six numbers will be drawn and I have made an assumption that these rounds are not independent and that each round is related to its previous round. So I am looking for a prediction algorithm that will try learn about this and output one or more predictions.

This will be a supervised learning for the algorithm. Using the above example data, it will be pre-processed so that it becomes:

[
 [[3,10,16,19,34,45], [7,14,15,20,28,41]],
 [[7,14,15,20,28,41], [2,6,18,24,30,37]],
 ...
]

Obviously, for each tuple, input is on the left while output is on the right. Also, I have another assumption that the numbers printed on the balls are really just labels, so the algorithm should consider them as categorical rather than numerical attribute. And there is no ordering for these labels, though the data appears sorted numerically in ascending order. So when the algorithm tries to learn from the training data, it should not be affected by the ball ordering in any way.

Finally, after it has been trained, if I input, for example [3,5,6,11,34,42], it should output a list of the algorithm's best guess predictions.

So what is the best prediction algorithm and the recommended programming tool for such task?

EDIT: In theory, current round should be unrelated to previous round. But I still think they do have some correlation (though very little), which is not obvious and hard to explain.


texts: What's an appropriate datastore for variable length sequence data for PyTorch consumption?

texts:
I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats.

I need a datastore that can rapidly serve these up in batches for PyTorch training. I also need to be able to rapidly write new sequences to the store. It's like an experience replay buffer for reinforcement learning, but I want to store every single run.

These sequences should each have some attached structured metadata in a queryable format so that I can select subsets of sequences.

The best solution looks like HDF5 - either through h5py or PyTables - except that I don't know how to make it efficiently handle the variable sequence lengths. Padding isn't appropriate because of the wildly varying lengths, and storing each sequence as its own HDF5 dataset seems like a poor idea as HDF5 doesn't seem to be optimised for massive numbers of small datasets.

Ideas on my radar include Pandas multi-indexing, HDF5 region references, and building a custom metadata index system from scratch. I'm not really sure where to go from here.

Storage compactness matters - I need to be reasonably efficient with my storage space.


texts:
I want to build a classification model to match customers and products. I have a description of each product, and a description of each customer, and the label : customer *i* buy/did not buy product *j*.
Each sample/row is a pair (customer, product), so Feature 1 is customer's description, Feature 2 is product's description, and the target variable y is: "y = 1 : customer buys product", "y = 0 otherwise". The goal is to predict for new arriving products whether each customer is going to buy them or not.
I want to use Tf-Idf Vectorizer. I don't in which specific step I should fit_transform the descriptions, and how to put together Feature 1 with Feature 2.

Should I concatenate the descriptions of each pair (customer, product) and fit_transform only once I have the concatenation?

Should I put together 2 columns using ColumnTransformer? If so, is the classifier going to fit correctly the obtained features?

Should I transform using a unique vocabulary?

I found here a reference of three possible ways of working with two columns, but I don't see which one fits for my case.
Ps. Until now, I only got to build a similarity pairwise coefficient (using this), but there is no classification, and I know using labelled data can help. In particular, similarity measure gives the same weight to any text coincidence, but some coincidences should be more important than others.


texts: How does a Q algorithm consider future rewards?

texts:
I am trying to understand the underlying logic of Q learning (deep Q learning to be precise). At the moment I am stuck at the notion of future rewards.

To understand the logic, I am reviewing some of the present code samples. This one seemed quite interesting, so I went through it:

https://github.com/keon/deep-q-learning/blob/master/dqn.py

Here is the gist of the code that does the actual training of the underlying deep neural network:

def replay(self, batch_size):
    minibatch = random.sample(self.memory, batch_size)
    for state, action, reward, next_state, done in minibatch:
        target = reward
        if not done:
            target = (reward + self.gamma *
                      np.amax(self.model.predict(next_state)[0]))
        target_f = self.model.predict(state)
        target_f[0][action] = target
        self.model.fit(state, target_f, epochs=1, verbose=0)
    if self.epsilon > self.epsilon_min:
        self.epsilon *= self.epsilon_decay

In the 5th line of the code, (after the if not done line) we are adding the discounted reward of the next step, to the present step, and setting it as the target reward of the executed action to be trained. So, the way I see it, we have the reward of the executed action, and discounted possible reward of the following action, combined.

As far as I understand, in each iteration, Q-learning algorithm predicts the future reward of next step (and next step only) using the machine learning technique in use (be it the CNN, DNN etc.). And we are multiplying the reward of next step (and that specific next step only) with discount rate, to make it less important than the immediate reward (with the ratio we specified). So, my question is, how does the algorithm takes even further steps (say, 5 steps) ahead into account?


texts: What are some ways to train, test and render an RL autonomous drone in a simulator?

texts:
I completed the Udacity nanodegree in Deep Learning but found the final project to be extremely difficult since the only visual feedback were from plots via matplotlib (rotor speeds, x/y/z coordinates, x/y/z velocity, etc).  Is there any way to render the learning per episode as done in openai gym?

A quick search for Python game engines only confused me more in terms of which one to use.

Here is a link to the final project, ie. fly a quadcopter using DDPG algorithm:  https://github.com/udacity/RL-Quadcopter-2

It has a physics_sim.py, task.py, agent.py and a main ipynb notebook.


texts: Intuition for the regularization parameter in SVM

texts: How does varying the regularization parameter in an SVM change the decision boundary for a non-separable dataset? A visual answer and/or some commentary on the limiting behaviors (for large and small regularization) would be very helpful.


texts:
My Keras LSTM NN improves to a val_loss of around 0.005 when training with 75% of my 4000-s long time series. I've ran the GridSearchCV to tune the hyperparameters but when I try to predict the remaining 25% of the series I get unsatisfactory results. The predictions lose quality in its very first points.

The data is MinMax-scaled and I haven't bothered re-scaling yet since it wouldn't matter for now.
Does it mean the LSTM is not adequate for my case? Is there anything else I could try?
My time series comes from a dynamic FEM simulation with time step of 0.1 s, ie, two (or even more) consecutive points are very similar. To avoid redundant data I filtered the series to work with steps of 1 s.
I've also tried to use tensorflow.keras.preprocessing.sequence.TimeseriesGenerator but it starts training the NN for a few moments and then complains about the dimensions of the input (I'm trying to solve the issue on StackOverflow, but any help would be greatly appreciated). Regardless, I really don't see why this function would be important since my series is already in a y=f(x) kind of structure.


texts:
I have X_train that shapes (14599, 13), i'm trying to impute NaN with column's median but somehow it imputes with row resulting error because in a row there are date, and other than integer values. I already lookup if SimpleImputer has axis parameter but could not find that it exists. How to solve this?

import pandas as pd
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split

plt.close('all')
avo_sales = pd.read_csv('avocados.csv')
avo_sales.rename(columns = {'4046':'small PLU sold',
                            '4225':'large PLU sold',
                            '4770':'xlarge PLU sold'},
                 inplace= True)
avo_sales.columns = avo_sales.columns.str.replace(' ','')

plt.scatter(avo_sales.Date,avo_sales.TotalBags)

x = np.array(avo_sales.drop(['TotalBags'],1))
y = np.array(avo_sales.TotalBags)

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)

imp = SimpleImputer(strategy='median')
X_train = imp.fit_transform(X_train)

the output

ValueError: Cannot use median strategy with non-numeric data:
could not convert string to float: '12/31/2017'
```


texts: Autoencoder with specific features in encoded layer

texts:
I want to create an autoencoder which will map date + 24 hourly data

for example 25/08/2020 - [25,8,2020,23,3,11,31,43,43,23,23,45,23,11,32,43,52,34,23,12,43,52,32,54,23,54,34]

to encoded layer of 3 features [day, month, year] so that I can predict hourly values out of these.
I am using Keras and I tried different tutorials but I don't know how to "focus" the model on these date features so later on I could receive an hourly data out of these. Can you please give me hint how can I do this?


texts: How does Google categorize results from its image search?

texts:
While doing a Google image search, the page displays some figured out categories for the images of the topic being searched for. I'm interested in learning how this works, and how it chooses and creates categories.

Unfortunately, I couldn't find much about it at all. Is anyone able to shed some light on algorithms they may be using to do this, and what basis these categories are created from?

For example, if I search for "animals" I get the categories:

  "cute", "baby", "wild", "farm", "zoo", "clipart".

If I go into "wild", I then have subcategories:

  "forest", "baby", "africa", "clipart", "rainforest", "domestic".


texts: Tool to Label Images for Supervised Classification

texts:
I have a couple thousand photos of whales taken from drones and I'm planning to build a simple binary classifier to run on these and future images to see if they contain a whale. I'd like to label specific pixels within the image as whale (1) or not whale (0) and I'm at a loss for a good free tool to do this. I'm imagining something where I can just zoom in and manually draw an outline around a whale but I'm open to other options. Does anyone know of a good way to do this?

Thanks!


texts: Spatial and temporal information processing together (CNN and LSTM)

texts:
I have small problem that requires to process both spatial and temporal information.

I need to predict vehicle's trajectory based on previous trajectory information and map information. My current situation:

Predicting future trajectory of vehicle was successfully done using seq2seq model. Where I give previous N points and model returns K possible trajectories
I have tried crop map information(based on vehicle's position) and giving it to CNNLSTM model with trajectory information. But even test accuracy was not good.

  What architecture is best for my problem?

P.S I am new to this forum. Feel free to improve question or point out my mistakes.


texts: Why is my neural net getting a poor accuracy?

texts:
EPOCH=40
batch_size=50
mnist=input_data.read_data_sets("MNIST_data", one_hot=True)

tf.reset_default_graph()

input_X = tf.placeholder(tf.float32, shape=[None, 784])
input_y = tf.placeholder(tf.int64, shape=[None, 10])

input_layer=tf.layers.dense(input_X, 784, activation=tf.nn.sigmoid)
hidden1=tf.layers.dense(input_layer, 256, activation=tf.nn.sigmoid)
hidden2=tf.layers.dense(hidden1, 256, activation=tf.nn.sigmoid)
output=tf.layers.dense(hidden2, units=10)
output=tf.nn.softmax(output)

entropy=tf.nn.softmax_cross_entropy_with_logits(labels=input_y, logits=output)
loss=tf.reduce_mean(entropy)
step=tf.train.GradientDescentOptimizer(0.01).minimize(loss)

#correct=tf.nn.in_top_k(tf.argmax(output, y, 1) rank error

correct=tf.equal(tf.argmax(output, 1), tf.argmax(input_y, 1))
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

sess=tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

for epoch in range(EPOCH):
  for i in range(1000):
    X, y=mnist.train.next_batch(batch_size)
    sess.run(loss, feed_dict={input_X:X, input_y:y})
  acc_train=accuracy.eval(feed_dict={input_X:X, input_y:y})
  acc_val=accuracy.eval(feed_dict={input_X: mnist.validation.images
                                  , input_y: mnist.validation.labels})
  print(epoch, "Train accuracy: ", acc_train, "\n Val accuracy: ", acc_val)

Results went something like this-

0 Train accuracy:  0.1
 Val accuracy:  0.0986
1 Train accuracy:  0.14
 Val accuracy:  0.0986
2 Train accuracy:  0.06
 Val accuracy:  0.0986
3 Train accuracy:  0.06
 Val accuracy:  0.0986

I have referenced part of the code from O'Reily textbook on Tensorflow.

EDIT- found this O' Reily guide online-https://github.com/ageron/handson-ml/blob/master/10_introduction_to_artificial_neural_networks.ipynb

They have a code similar to mine but their's works just fine.


texts: Machine-learning algorithm / library for outcome prediction

texts: I am extremely new to this data science world so bear with me if my question is not very clear, I'd be glad to clarify. What I am looking for is simple: train a program with a set of values (5 ordered integer inputs, 1 boolean output). Then I would give it 5 inputs where the outcome is not known, and it has to tell me the outcome.


texts: Why continuous features are more important than categorical features in decision tree models?

texts:
I have both categorical and continuous features in my prediction model and want to select (and rank) most important features.

I have converted all categorical variables into dummy variables using one hot encoding (for better interpretation in my logistic regression model).

On one hand, I use LogisticRegression (sklearn) and rank the most significant features by using their coefficients. In this way, I see both categorical and continuous variables among the most important features.

On the other hand, When I want to rank the features by using Decision Tree models (SelectFromModel) they always give higher scores (feature_importances_) first to continuous features and then to categorical (dummy) variables. A completely different behavior in comparison with Logistic Regression.

Whilst the performance of Decision Tree models is much higher (about 15%) than the performance of Logistic Regression, I want to know which sorting of features (Decision Tree or Logistic Regression) is more correct? And why Decision Tree models give more priority to continuous features?


texts: How to divide a dataset for training and testing when the features and targets are in two different files?

texts:
I am trying to divide a dataset into training dataset and testing dataset for multi-label classification. The datset I am working on is this one. It is divided into a file which contains the features and another file which contains the targets. They look like this below.

This is the image about the features.

This is the image about the targets.

I intend to use this dataset for multilabel classification. I am following this tutorial . Here the dataset looks like this.

The dataset that I am working on has 17203824 samples and 58255 different and unique labels in the target file. So to follow the tutorial what I intend to create is a new numpy 2d array with 17203824 rows and 58255 columns where appropriate indices will be marked with 1. I am able to create it. But when I try to populate with 1s in the appropriate indices, I am getting an error. Its says that I don't have enough memory. My code is given below.

questions = pd.read_csv("/kaggle/input/stacklite/questions.csv")
question_tags = pd.read_csv("/kaggle/input/stacklite/question_tags.csv")
d = {v: i[0] for i, v in np.ndenumerate(question_tags["Tag"].unique())}

y = np.zeros([questions.shape[0], len(question_tags["Tag"].unique())], dtype = int)

for k in question_tags["Tag"]:
    j = d[k]
    for i, l in enumerate(y):
        y[i][j] = 1

Can anyone please help in telling me how I should proceed?


texts: Instead of one-hot encoding, can I store the same information in one column using a single value?

texts:
Rather than creating 15 additional columns full of sparse binary data, could I:

1) use the first 15 prime numbers as indexes for the 15 categories

2) store data by multiplying the prime numbers of the categories that otherwise would have a value of 1 in one-hot encoding

3) retrieve data by factorizing the value generated by multiplying unique prime numbers

Ex: 1914 would yield the list [2, 3, 11, 29] which would let you know that the user with the 1914 value has property 2, 3, 11, and 29 but nothing else.

I understand this is limited because BIGINTs can only hold the product of the first 15 prime numbers, but would it not still be useful in some situations and save time when searching the database? The entire table would be 14 columns smaller. I guess this is less about machine learning algorithms and more about storing and retrieving data.


texts: Is RL applied to animal dispersion a valid approach?

texts:
I have an agent which has a medium-sized, discrete set of actions $A$: $10<|A|<100$. The actions can be taken over an infinite horizon of 1 second per timestep $t$. The world is essentially pictures from a static camera and the states $S$ are the amount of animals detected on the picture (assuming perfect detection for simplicity). We can safely say that $max(S) < 200$. Each action $a$ is meant to lower the amount of detections because of dispersion or fear from the animals. The reward is the difference of detections from $s_{-1}$ to $s$.

Admittedly I am only a beginner in RL, so I haven't seen much more than MDP's. I believe that this isn't a Markov problem since the states aren't independent (past actions having an impact on the outcome of the current action). That being said, I'm wondering if there's a specific RL algorithm for this setup or if RL even is the right way to go ?


texts:
I have studied the Perceptron algorithm and now I'm trying to understand logistic regression. I know what odd ratio and Logit functions are but I don't understand why we create the following equation:

$logit(\frac{p}{1-p}) = \bar{\omega}^{T}\bar{x}$

where w is the weight vector. What does it represent, and why?

Can anyone show me, how does that work?


texts: How to model a decimal response between 0 to 1 with a GLM in R

texts:
I am trying to model a response variable which is a proportion (so a response between 0 and 1, see picture for distribution).

Ideally I would like to model it without using the actual counts, so as a decimal.

So far I have been using a binomial family in R.

model <- glm(Response ~
                X1 +
                X2 +
                X3,
              data = Training_data,
              family = 'binomial')

I think the model is doing okay, but when I use it for predictions it doesn't do a good job predicting when the ratio is 1 (As you can see from the picture).

I'm not sure if my approach of using a binomial distribution is wrong?

Thanks for your help ]1


texts: Is it possible to determine the correlation direction of two variables using factor loadings?

texts:
Specifically, I am wondering if you can determine this when each factor explains only one of the variables well.

Here's an illustration of a factor pattern to clarify:

Is it possible to say that variable 1 and variable 3 are positively correlated? If not, what other information would be needed?


texts: Can we calculate AUC for deep learning based regression task

texts:
There is  a paper [ref.attaced], where they used a deep  learning based regression and evaluated using mse and AUC. The targets are continuous values such as 1,2,3 up to 16 and has been normalized before training.
Is it possible to calculate  the AUC for regression task.?


texts: In elbow curve how to find the point from where the curve starts to rise?

texts:
I am computing a distance metric on my data. The result is then being sorted in ascending order.
The samples having distance more than a specific threshold are to be marked as outliers and will be discarded. Below is a plot of all distance values.

As evident from the graph, after a certain point, the graph rises quite rapidly and even the datapoints get sparse. I need to calculate that point from where this happens and mark that point as threshold value.


texts: If I do not do any data normailization, is datagen.fit required in Keras?

texts:
I use keras for training an image classification problem as follows:

datagen = ImageDataGenerator(
    featurewise_center=False,
    featurewise_std_normalization=False,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True)

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(x_train)

# alternative to model.fit_generator
for e in range(epochs):
    print('Epoch', e)
    batches = 0
    for x_batch, y_batch in datagen.flow(x_train, y_train, batch_size=32):
        model.fit(x_batch, y_batch)
        batches += 1
        if batches >= len(x_train) / 32:
            # we need to break the loop by hand because
            # the generator loops indefinitely
            break

I would like to know if datagen.fit is required if I do not apply any data normalization?